[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:05 dist-test-slave-npg2 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:05 dist-test-slave-npg2 krb5kdc[399](info): set up 2 sockets
Sep 24 04:41:05 dist-test-slave-npg2 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:41:06.667733 392 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:06 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:06.799526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40475
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:06.915835 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:06.916179 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:06.916265 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:06.920467 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:06.920572 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:06.920617 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:06.920665 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:06.920697 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:06.926211 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:06.927497 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:06.928614 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:06.938050 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:06.938078 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:06.938144 412 server_base.cc:1034] running on GCE node
W20260924 04:41:06.938014 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:06.938992 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:06.940322 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:06.941547 412 hybrid_clock.cc:648] HybridClock initialized: now 1790224866941512 us; error 56 us; skew 500 ppm
Sep 24 04:41:06 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:06.944782 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:06.945871 412 webserver.cc:460] Webserver started at http://127.0.98.62:34305/ using document root <none> and password file <none>
I20260924 04:41:06.946492 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:06.946590 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:06.946832 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:06.949048 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "bc157327de8e4334b6817c7506405de3"
format_stamp: "Formatted at 2026-09-24 04:41:06 on dist-test-slave-npg2"
server_key: "8337eb27578078eb48d42dd8b44d2da6"
server_key_iv: "974b08ebb12b1fdca0b1f2375a10d572"
server_key_version: "encryptionkey@0"
I20260924 04:41:06.949697 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc157327de8e4334b6817c7506405de3"
format_stamp: "Formatted at 2026-09-24 04:41:06 on dist-test-slave-npg2"
server_key: "8337eb27578078eb48d42dd8b44d2da6"
server_key_iv: "974b08ebb12b1fdca0b1f2375a10d572"
server_key_version: "encryptionkey@0"
I20260924 04:41:06.953832 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 04:41:06.956645 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:06.957960 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:06.958266 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "bc157327de8e4334b6817c7506405de3"
format_stamp: "Formatted at 2026-09-24 04:41:06 on dist-test-slave-npg2"
server_key: "8337eb27578078eb48d42dd8b44d2da6"
server_key_iv: "974b08ebb12b1fdca0b1f2375a10d572"
server_key_version: "encryptionkey@0"
I20260924 04:41:06.958400 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:06.977913 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:06.981676 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:06.981917 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:06.989620 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40475
I20260924 04:41:06.989645 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40475 every 8 connection(s)
I20260924 04:41:06.990649 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:06.993723 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:06.996646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 412
I20260924 04:41:06.996780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:06.998140 392 external_mini_cluster.cc:1428] Setting key a91dc10d7daa52c162fe07f29e67078c
I20260924 04:41:06.999737 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3: Bootstrap starting.
I20260924 04:41:07.002692 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:07.003588 480 log.cc:824] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:07.005610 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3: No bootstrap required, opened a new log
I20260924 04:41:07.009248 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc157327de8e4334b6817c7506405de3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260924 04:41:07.009471 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:07.009545 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc157327de8e4334b6817c7506405de3, State: Initialized, Role: FOLLOWER
I20260924 04:41:07.010032 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc157327de8e4334b6817c7506405de3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260924 04:41:07.010164 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:07.010253 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:07.010375 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:07.011400 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc157327de8e4334b6817c7506405de3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260924 04:41:07.011750 480 leader_election.cc:302] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc157327de8e4334b6817c7506405de3; no voters:
I20260924 04:41:07.012140 480 leader_election.cc:288] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:07.012344 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:07.012578 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [term 1 LEADER]: Becoming Leader. State: Replica: bc157327de8e4334b6817c7506405de3, State: Running, Role: LEADER
I20260924 04:41:07.012885 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc157327de8e4334b6817c7506405de3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260924 04:41:07.013587 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:07.014955 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc157327de8e4334b6817c7506405de3. Latest consensus state: current_term: 1 leader_uuid: "bc157327de8e4334b6817c7506405de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc157327de8e4334b6817c7506405de3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } } }
I20260924 04:41:07.015075 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:07.015378 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc157327de8e4334b6817c7506405de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc157327de8e4334b6817c7506405de3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } } }
I20260924 04:41:07.015478 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:07.015592 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:07.018913 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:07.023980 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.001461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.62:40475)
0924 04:41:07.001934 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:07.001950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:07.003043 (+ 1093us) server_negotiation.cc:407] Connection header received
0924 04:41:07.006390 (+ 3347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:07.006408 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:07.006725 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:07.007057 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:07.009332 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.010610 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.011988 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.012710 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.014321 (+ 1611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.018049 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:07.018055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:07.018069 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:07.018103 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:07.020035 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.020605 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.020611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.020617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.020743 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.021276 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.021279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.021281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.021766 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:07.021974 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:07.022668 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 04:41:07.022953 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":164,"threads_started":1}
I20260924 04:41:07.025753 494 catalog_manager.cc:1347] Generated new cluster ID: cc4af3de3b8f455ab6481d775a2a7336
I20260924 04:41:07.025818 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:07.031534 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:07.032622 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:07.042498 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc157327de8e4334b6817c7506405de3: Generated new TSK 0
I20260924 04:41:07.043249 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:07.078480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:07.192785 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:07.193042 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:07.193087 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:07.197260 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:07.197322 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:07.197383 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:07.202621 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:07.203701 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:07.204720 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:07.214475 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:07.214607 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:07.214637 506 server_base.cc:1034] running on GCE node
W20260924 04:41:07.214488 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:07.215245 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:07.215927 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:07.217087 506 hybrid_clock.cc:648] HybridClock initialized: now 1790224867217070 us; error 31 us; skew 500 ppm
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:07.220086 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:07.220979 506 webserver.cc:460] Webserver started at http://127.0.98.1:36281/ using document root <none> and password file <none>
I20260924 04:41:07.221509 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:07.221570 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:07.221740 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:07.223835 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "92745991348448149265091e8c3a1278"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "ec30c75d6dfc3aec278f9da3650e4232"
server_key_iv: "2ed232a392f9e997a7ebfc3637c41dcc"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.224354 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92745991348448149265091e8c3a1278"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "ec30c75d6dfc3aec278f9da3650e4232"
server_key_iv: "2ed232a392f9e997a7ebfc3637c41dcc"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.228386 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:07.230999 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.232342 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:07.232492 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "92745991348448149265091e8c3a1278"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "ec30c75d6dfc3aec278f9da3650e4232"
server_key_iv: "2ed232a392f9e997a7ebfc3637c41dcc"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.232613 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:07.245738 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:07.249554 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:07.249724 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:07.250263 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:07.251065 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:07.251185 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.251250 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:07.251271 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.260080 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34143
I20260924 04:41:07.260192 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34143 every 8 connection(s)
I20260924 04:41:07.261063 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:07.265092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 506
I20260924 04:41:07.265179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:07.265520 392 external_mini_cluster.cc:1428] Setting key c61aed7747d610c60da5b7894f246818
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:07.276584 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.263104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54847 (local address 127.0.98.62:40475)
0924 04:41:07.263286 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:07.263290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:07.264369 (+ 1079us) server_negotiation.cc:407] Connection header received
0924 04:41:07.266046 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:07.266049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:07.266114 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:07.266235 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:07.267850 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.268458 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.269681 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.269911 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.270028 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.272587 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:07.272591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:07.272593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:07.272622 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:07.274154 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.274815 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.274819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.274821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.274907 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.275354 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.275357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.275359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.275550 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:07.275663 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:07.276244 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 04:41:07.276372 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:07.277523 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.263567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40475 (local address 127.0.98.1:54847)
0924 04:41:07.264006 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:07.264185 (+ 179us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:07.265793 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:07.266450 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:07.266458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:07.266864 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:07.267687 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:07.267702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.268627 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.268629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:07.269534 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:07.269554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.270068 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.270072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:07.270260 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.270885 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:07.270901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:07.272413 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:07.274321 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:07.274327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:07.274352 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:07.274702 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:07.275012 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:07.275015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:07.275017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:07.275248 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:07.275674 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:07.275684 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:07.275985 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 04:41:07.276185 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 04:41:07.276540 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20260924 04:41:07.278962 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40475
I20260924 04:41:07.279244 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:07.279767 635 heartbeater.cc:511] Master 127.0.98.62:40475 requested a full tablet report, sending...
I20260924 04:41:07.281333 444 ts_manager.cc:194] Registered new tserver with Master: 92745991348448149265091e8c3a1278 (127.0.98.1:34143)
I20260924 04:41:07.282486 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:07.295524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:07.409140 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:07.409407 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:07.409451 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:07.413609 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:07.413677 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:07.413739 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:07.419735 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:07.420756 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:07.421844 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:07.431885 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:07.431945 640 server_base.cc:1034] running on GCE node
W20260924 04:41:07.431872 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:07.431872 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:07.432600 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:07.433265 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:07.434469 640 hybrid_clock.cc:648] HybridClock initialized: now 1790224867434432 us; error 49 us; skew 500 ppm
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:07.437739 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:07.438616 640 webserver.cc:460] Webserver started at http://127.0.98.2:42459/ using document root <none> and password file <none>
I20260924 04:41:07.439220 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:07.439288 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:07.439486 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:07.441474 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "be9e5db15801442e936863dddae0acee"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "e27e5d59911cfa14ec8046d17394b7c8"
server_key_iv: "ebc9333d51869a8aeec7e82620d1e335"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.441982 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be9e5db15801442e936863dddae0acee"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "e27e5d59911cfa14ec8046d17394b7c8"
server_key_iv: "ebc9333d51869a8aeec7e82620d1e335"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.445791 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 04:41:07.448217 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.449489 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:07.449592 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "be9e5db15801442e936863dddae0acee"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "e27e5d59911cfa14ec8046d17394b7c8"
server_key_iv: "ebc9333d51869a8aeec7e82620d1e335"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.449682 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:07.467315 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:07.470642 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:07.470800 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:07.471424 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:07.472225 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:07.472272 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.472313 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:07.472329 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.482515 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42061
I20260924 04:41:07.482534 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42061 every 8 connection(s)
I20260924 04:41:07.483533 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:07.492302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 640
I20260924 04:41:07.492461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:07.492786 392 external_mini_cluster.cc:1428] Setting key c8547773bb36d03ec6aa6cfb59be9de2
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:07.498340 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.485480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34247 (local address 127.0.98.62:40475)
0924 04:41:07.485628 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:07.485633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:07.486568 (+ 935us) server_negotiation.cc:407] Connection header received
0924 04:41:07.487941 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:07.487944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:07.487990 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:07.488074 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:07.489481 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.489992 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.491295 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.491512 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.491594 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.494665 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:07.494668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:07.494670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:07.494692 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:07.496170 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.496714 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.496717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.496721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.496805 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.497157 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.497163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.497168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.497400 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:07.497516 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:07.498044 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 04:41:07.498166 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:41:07.499276 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.485953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40475 (local address 127.0.98.2:34247)
0924 04:41:07.486407 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:07.486450 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:07.487687 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:07.488408 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:07.488416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:07.488798 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:07.489343 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:07.489357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.490112 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.490114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:07.491091 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:07.491101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.491642 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.491645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:07.491794 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.492473 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:07.492497 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:07.494500 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:07.496307 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:07.496313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:07.496330 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:07.496600 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:07.496908 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:07.496911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:07.496912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:07.497057 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:07.497513 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:07.497520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:07.497790 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 04:41:07.497995 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 04:41:07.498355 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260924 04:41:07.500451 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40475
I20260924 04:41:07.500670 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:07.501163 769 heartbeater.cc:511] Master 127.0.98.62:40475 requested a full tablet report, sending...
I20260924 04:41:07.502278 444 ts_manager.cc:194] Registered new tserver with Master: be9e5db15801442e936863dddae0acee (127.0.98.2:42061)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:41:07.503063 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:07.522466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:07.639196 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:07.639467 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:07.639519 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:07.643579 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:07.643635 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:07.643695 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:07.649142 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:07.650190 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:07.651204 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:07.661906 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:07.661936 774 server_base.cc:1034] running on GCE node
W20260924 04:41:07.661906 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:07.662120 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:07.662642 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:07.663448 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:07.664630 774 hybrid_clock.cc:648] HybridClock initialized: now 1790224867664617 us; error 31 us; skew 500 ppm
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:07.667711 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:07.668705 774 webserver.cc:460] Webserver started at http://127.0.98.3:36309/ using document root <none> and password file <none>
I20260924 04:41:07.669353 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:07.669453 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:07.669682 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:07.672017 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f4cc4a3d0ea4b9187fc534505ab477f"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "3d3fedacb82328011131348c22499410"
server_key_iv: "46b0b11a5d0df2395649a39960082992"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.672631 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f4cc4a3d0ea4b9187fc534505ab477f"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "3d3fedacb82328011131348c22499410"
server_key_iv: "46b0b11a5d0df2395649a39960082992"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.676663 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:07.679322 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.680625 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:41:07.680779 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "3f4cc4a3d0ea4b9187fc534505ab477f"
format_stamp: "Formatted at 2026-09-24 04:41:07 on dist-test-slave-npg2"
server_key: "3d3fedacb82328011131348c22499410"
server_key_iv: "46b0b11a5d0df2395649a39960082992"
server_key_version: "encryptionkey@0"
I20260924 04:41:07.680902 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:07.695266 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:07.698530 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:07.698765 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:07.699478 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:07.700330 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:07.700402 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.700460 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:07.700510 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:07.710533 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40375
I20260924 04:41:07.710551 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40375 every 8 connection(s)
I20260924 04:41:07.711583 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:07.720184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 774
I20260924 04:41:07.720323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:07.720551 392 external_mini_cluster.cc:1428] Setting key 1715c7869209022b3b1b1ea60863be3a
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:07.726867 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.713378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37125 (local address 127.0.98.62:40475)
0924 04:41:07.713550 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:07.713555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:07.714398 (+ 843us) server_negotiation.cc:407] Connection header received
0924 04:41:07.715931 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:07.715934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:07.715993 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:07.716103 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:07.717685 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.718230 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.719457 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.719676 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.719797 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.723010 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:07.723016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:07.723019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:07.723049 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:07.724699 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.725332 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.725336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.725338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.725420 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.725772 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.725777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.725779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.725963 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:07.726072 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:07.726582 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 04:41:07.726709 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:41:07.727788 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.713780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40475 (local address 127.0.98.3:37125)
0924 04:41:07.714249 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:07.714293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:07.715695 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:07.716287 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:07.716295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:07.717044 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:07.717534 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:07.717548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.718360 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.718364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:07.719326 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:07.719335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.719837 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.719841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:07.720044 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.720854 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:07.720875 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:07.722840 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:07.724838 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:07.724845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:07.724859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:07.725216 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:07.725523 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:07.725527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:07.725529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:07.725670 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:07.726070 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:07.726077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:07.726342 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 04:41:07.726541 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:41:07.726883 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":146,"threads_started":1}
I20260924 04:41:07.728967 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40475
I20260924 04:41:07.729188 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:07.729692 903 heartbeater.cc:511] Master 127.0.98.62:40475 requested a full tablet report, sending...
I20260924 04:41:07.730756 444 ts_manager.cc:194] Registered new tserver with Master: 3f4cc4a3d0ea4b9187fc534505ab477f (127.0.98.3:40375)
I20260924 04:41:07.731367 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37125
I20260924 04:41:07.735343 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:07.762682 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.752355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.1:34143)
0924 04:41:07.752639 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:07.752653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:07.752678 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:41:07.752810 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:07.752816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:07.753015 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:07.753137 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:07.754046 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.754843 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.755805 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.756168 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.756267 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.758853 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:07.758861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:07.758875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:07.758898 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:07.760718 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.761141 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.761147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.761148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.761233 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.761531 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.761533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.761535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.761796 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:07.761949 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:07.762220 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 04:41:07.762487 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
W20260924 04:41:07.763828 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34938
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:07.790995 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:07.781819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.98.1:34143)
0924 04:41:07.782022 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:07.782026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:07.782129 (+ 103us) server_negotiation.cc:407] Connection header received
0924 04:41:07.782250 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:07.782253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:07.782298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:07.782378 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:07.783110 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.783789 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.784663 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:07.784898 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:07.785060 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:07.787524 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:07.787527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:07.787529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:07.787588 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:07.789218 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.789691 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.789694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.789695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.789774 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:07.790099 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:07.790103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:07.790104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:07.790283 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:07.790395 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:07.790654 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 04:41:07.790792 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 04:41:07.792133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 506
I20260924 04:41:07.798172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 640
I20260924 04:41:07.806716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 774
I20260924 04:41:07.812587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 412
2026-09-24T04:41:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2507 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[917](info): set up 2 sockets
Sep 24 04:41:07 dist-test-slave-npg2 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:41:08.867079 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:08 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:08.995484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:09.116742 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:09.117151 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:09.117388 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:09.121778 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:09.121891 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:09.121933 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:09.121968 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:09.121999 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:09.127682 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:09.128846 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:09.129928 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:41:09.138615 930 server_base.cc:1034] running on GCE node
W20260924 04:41:09.138605 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.138561 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.138566 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:09.139480 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:09.140586 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:09.141814 930 hybrid_clock.cc:648] HybridClock initialized: now 1790224869141802 us; error 36 us; skew 500 ppm
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:09.144668 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:09.145533 930 webserver.cc:460] Webserver started at http://127.0.98.62:35867/ using document root <none> and password file <none>
I20260924 04:41:09.146093 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:09.146185 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:09.146473 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:09.148571 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "0998bb7bb0fe4ec18b8533486a5b4735"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "46e9043d25689f072f8ffd319b34205d"
server_key_iv: "063c8bf9a601606c32242700a7aa2750"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.149137 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "0998bb7bb0fe4ec18b8533486a5b4735"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "46e9043d25689f072f8ffd319b34205d"
server_key_iv: "063c8bf9a601606c32242700a7aa2750"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.153012 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:09.155551 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.156877 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:41:09.157048 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "0998bb7bb0fe4ec18b8533486a5b4735"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "46e9043d25689f072f8ffd319b34205d"
server_key_iv: "063c8bf9a601606c32242700a7aa2750"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.157248 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:09.170704 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:09.174052 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:09.174276 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:09.181691 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40523
I20260924 04:41:09.181721 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40523 every 8 connection(s)
I20260924 04:41:09.182736 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:09.185732 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:09.191565 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735: Bootstrap starting.
I20260924 04:41:09.192406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 930
I20260924 04:41:09.192498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:09.192764 392 external_mini_cluster.cc:1428] Setting key 6cc32e170f42b52d05a5d71bb11e0a77
I20260924 04:41:09.194397 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:09.195183 998 log.cc:824] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:09.197079 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735: No bootstrap required, opened a new log
I20260924 04:41:09.200788 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260924 04:41:09.201074 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:09.201143 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0998bb7bb0fe4ec18b8533486a5b4735, State: Initialized, Role: FOLLOWER
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:09.201642 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260924 04:41:09.201818 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:09.201893 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:09.202008 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:09.203277 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260924 04:41:09.203670 998 leader_election.cc:302] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0998bb7bb0fe4ec18b8533486a5b4735; no voters:
I20260924 04:41:09.204022 998 leader_election.cc:288] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:09.204174 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:09.204411 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [term 1 LEADER]: Becoming Leader. State: Replica: 0998bb7bb0fe4ec18b8533486a5b4735, State: Running, Role: LEADER
I20260924 04:41:09.204788 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260924 04:41:09.205421 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:09.206502 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } } }
I20260924 04:41:09.206636 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:09.206952 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0998bb7bb0fe4ec18b8533486a5b4735. Latest consensus state: current_term: 1 leader_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0998bb7bb0fe4ec18b8533486a5b4735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } } }
I20260924 04:41:09.207043 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:09.207926 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:09.208402 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.193883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.98.62:40523)
0924 04:41:09.194326 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:09.194342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:09.194376 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:41:09.195042 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:09.195058 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:09.195494 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:09.195828 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:09.196585 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.197619 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.198472 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.198956 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.199134 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:09.202025 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:09.202032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:09.202050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:09.202086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:09.204272 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:09.204792 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:09.204798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:09.204804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:09.204938 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:09.205369 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:09.205372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:09.205373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:09.205896 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:09.206093 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:09.207029 (+ 936us) server_negotiation.cc:299] Negotiation successful
0924 04:41:09.207411 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
I20260924 04:41:09.210990 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:09.216956 1012 catalog_manager.cc:1347] Generated new cluster ID: 0f79a068c4d4485bbd8268340809d1b2
I20260924 04:41:09.217096 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:09.235646 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:09.236649 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:09.250957 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0998bb7bb0fe4ec18b8533486a5b4735: Generated new TSK 0
I20260924 04:41:09.251936 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:09.291978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40523
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:09.436203 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:09.436486 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:09.436569 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:09.440819 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:09.440887 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:09.440982 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:09.447445 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:09.448653 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:09.449836 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:09.460558 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.460618 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:09.460783 1024 server_base.cc:1034] running on GCE node
W20260924 04:41:09.460630 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:09.461408 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:09.462186 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:09.463351 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790224869463315 us; error 52 us; skew 500 ppm
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:09.466284 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:09.467245 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41951/ using document root <none> and password file <none>
I20260924 04:41:09.467818 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:09.467877 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:09.468046 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:09.470120 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "32cafc7ddf8646c6994a2220a36e5954"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "7ef1496643b7d9dc0b7ef827120ec720"
server_key_iv: "c52576ad91917b62b7b8cace3b0408ab"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.470661 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32cafc7ddf8646c6994a2220a36e5954"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "7ef1496643b7d9dc0b7ef827120ec720"
server_key_iv: "c52576ad91917b62b7b8cace3b0408ab"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.474821 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:41:09.477609 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.479079 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:09.479276 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "32cafc7ddf8646c6994a2220a36e5954"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "7ef1496643b7d9dc0b7ef827120ec720"
server_key_iv: "c52576ad91917b62b7b8cace3b0408ab"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.479408 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:09.502379 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:09.505995 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:09.506330 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:09.507045 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:09.508229 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:09.508289 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.508374 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:09.508409 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.519023 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42803
I20260924 04:41:09.519042 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42803 every 8 connection(s)
I20260924 04:41:09.520026 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:09.530400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1024
I20260924 04:41:09.530514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:09.530824 392 external_mini_cluster.cc:1428] Setting key 54db634c699df3f62154d20d3824ed0a
I20260924 04:41:09.534770 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.521991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52761 (local address 127.0.98.62:40523)
0924 04:41:09.522206 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:09.522210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:09.523069 (+ 859us) server_negotiation.cc:407] Connection header received
0924 04:41:09.524519 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:09.524522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:09.524582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:09.524663 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:09.525884 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.526425 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.527684 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.527935 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.528060 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:09.530940 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:09.530944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:09.530947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:09.530985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:09.532628 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:09.533174 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:09.533179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:09.533181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:09.533263 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:09.533643 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:09.533646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:09.533648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:09.533835 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:09.533950 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:09.534484 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 04:41:09.534607 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 04:41:09.535804 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.522412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40523 (local address 127.0.98.1:52761)
0924 04:41:09.522901 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:09.522945 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:09.524275 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:09.524846 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:09.524853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:09.525219 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:09.525718 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:09.525732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.526567 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.526571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:09.527550 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:09.527560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.528103 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.528107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:09.528279 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:09.528912 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:09.528928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:09.530706 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:09.532761 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:09.532768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:09.532786 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:09.533064 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:09.533364 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:09.533368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:09.533369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:09.533524 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:09.533949 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:09.533958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:09.534232 (+ 274us) client_negotiation.cc:769] Sending connection context
0924 04:41:09.534445 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 04:41:09.534814 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260924 04:41:09.537218 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40523
I20260924 04:41:09.537503 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:09.538213 1153 heartbeater.cc:511] Master 127.0.98.62:40523 requested a full tablet report, sending...
I20260924 04:41:09.539903 962 ts_manager.cc:194] Registered new tserver with Master: 32cafc7ddf8646c6994a2220a36e5954 (127.0.98.1:42803)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:09.541123 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:09.563443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40523
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:09.675227 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:09.675525 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:09.675617 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:09.679721 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:09.679816 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:09.679908 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:09.685474 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:09.686627 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:09.687728 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:41:09.697912 1158 server_base.cc:1034] running on GCE node
W20260924 04:41:09.697973 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.697947 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.698318 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:09.698722 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:09.699491 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:09.700682 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790224869700666 us; error 33 us; skew 500 ppm
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:09.703455 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:09.704416 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43243/ using document root <none> and password file <none>
I20260924 04:41:09.705004 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:09.705098 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:09.705355 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:09.707857 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "dc3a626addfd6ab0f36f1e3bbeed45d5"
server_key_iv: "75e395f425266fea2809316365d56dbb"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.708449 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "dc3a626addfd6ab0f36f1e3bbeed45d5"
server_key_iv: "75e395f425266fea2809316365d56dbb"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.712337 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:09.714912 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.716293 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:09.716439 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "dc3a626addfd6ab0f36f1e3bbeed45d5"
server_key_iv: "75e395f425266fea2809316365d56dbb"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.716557 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:09.742900 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:09.746315 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:09.746529 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:09.747321 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:09.748167 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:09.748241 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.748317 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:09.748351 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.757823 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38007
I20260924 04:41:09.757849 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38007 every 8 connection(s)
I20260924 04:41:09.758770 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:09.760403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1158
I20260924 04:41:09.760540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:09.760854 392 external_mini_cluster.cc:1428] Setting key f6104840f7d7409ad945341194c76fff
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:41:09.775395 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.760764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56455 (local address 127.0.98.62:40523)
0924 04:41:09.760920 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:09.760925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:09.761924 (+ 999us) server_negotiation.cc:407] Connection header received
0924 04:41:09.763960 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:09.763963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:09.764008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:09.764082 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:09.765378 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.765981 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.767981 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.768208 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.768293 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:09.771432 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:09.771436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:09.771438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:09.771462 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:09.773143 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:09.773716 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:09.773720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:09.773722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:09.773819 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:09.774266 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:09.774270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:09.774272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:09.774458 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:09.774572 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:09.775094 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 04:41:09.775232 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:41:09.776289 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.761211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40523 (local address 127.0.98.2:56455)
0924 04:41:09.761769 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:09.761825 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:09.763714 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:09.764311 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:09.764322 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:09.764711 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:09.765215 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:09.765229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.766136 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.766143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:09.767825 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:09.767839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:09.768379 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:09.768386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:09.768613 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:09.769272 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:09.769296 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:09.771253 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:09.773261 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:09.773267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:09.773283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:09.773550 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:09.773939 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:09.773945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:09.773950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:09.774129 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:09.774571 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:09.774580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:09.774848 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 04:41:09.775041 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:41:09.775463 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":134,"threads_started":1}
I20260924 04:41:09.777547 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40523
I20260924 04:41:09.777837 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:09.778360 1287 heartbeater.cc:511] Master 127.0.98.62:40523 requested a full tablet report, sending...
I20260924 04:41:09.779522 962 ts_manager.cc:194] Registered new tserver with Master: ab2ecbf125964fd9b7cd237d4f0c2bb2 (127.0.98.2:38007)
I20260924 04:41:09.780342 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:09.791071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40523
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:09.904965 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:09.905289 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:09.905373 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:09.909685 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:09.909787 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:09.909885 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:09.915266 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:09.916363 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:09.917412 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:09.927103 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.927090 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:09.927090 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:09.927428 1292 server_base.cc:1034] running on GCE node
I20260924 04:41:09.927913 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:09.928545 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:09.929736 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790224869929733 us; error 48 us; skew 500 ppm
Sep 24 04:41:09 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:09.932587 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:09.933475 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45513/ using document root <none> and password file <none>
I20260924 04:41:09.934037 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:09.934130 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:09.934365 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:09.936508 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "4c4b193b80d04f4199741e36cf75256f"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "fce8d3a3e7aa8562d1aeed52c0bc9625"
server_key_iv: "0a47b3bb8f7c225fb2fb0a6b8730f8e9"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.937081 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4c4b193b80d04f4199741e36cf75256f"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "fce8d3a3e7aa8562d1aeed52c0bc9625"
server_key_iv: "0a47b3bb8f7c225fb2fb0a6b8730f8e9"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.940925 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 04:41:09.943456 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.944954 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:41:09.945101 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "4c4b193b80d04f4199741e36cf75256f"
format_stamp: "Formatted at 2026-09-24 04:41:09 on dist-test-slave-npg2"
server_key: "fce8d3a3e7aa8562d1aeed52c0bc9625"
server_key_iv: "0a47b3bb8f7c225fb2fb0a6b8730f8e9"
server_key_version: "encryptionkey@0"
I20260924 04:41:09.945246 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:09.980860 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:09.984586 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:09.984807 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:09.985466 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:09.986409 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:09.986493 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.986565 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:09.986629 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:09.996294 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41535
I20260924 04:41:09.996320 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41535 every 8 connection(s)
I20260924 04:41:09.997252 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:09.997925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1292
I20260924 04:41:09.998046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:09.998356 392 external_mini_cluster.cc:1428] Setting key d6c2f989cd80af48fb84c778ea96bc0f
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:10.018002 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.999402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57651 (local address 127.0.98.62:40523)
0924 04:41:09.999520 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:09.999525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.000643 (+ 1118us) server_negotiation.cc:407] Connection header received
0924 04:41:10.001961 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.001968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.002023 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.002126 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:10.003964 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.005243 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.007453 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.007786 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.007884 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.011942 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:10.011947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:10.011950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:10.011981 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:10.014680 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.015559 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.015566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.015570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.015683 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.016324 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.016330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.016332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.016598 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:10.016718 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.017648 (+ 930us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.017805 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:41:10.019469 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:09.999942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40523 (local address 127.0.98.3:57651)
0924 04:41:10.000454 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:10.000533 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:10.001746 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:10.002374 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:10.002386 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:10.002944 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:10.003757 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.003773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.005418 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.005424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.007245 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.007257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.007964 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.007969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.008194 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.009213 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:10.009237 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:10.011724 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:10.014859 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:10.014869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:10.014891 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:10.015410 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:10.015805 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:10.015809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:10.015811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:10.016198 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:10.016740 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:10.016748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:10.017143 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 04:41:10.017397 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 04:41:10.017811 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260924 04:41:10.021186 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40523
I20260924 04:41:10.021476 1421 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:10.022111 1421 heartbeater.cc:511] Master 127.0.98.62:40523 requested a full tablet report, sending...
I20260924 04:41:10.023375 962 ts_manager.cc:194] Registered new tserver with Master: 4c4b193b80d04f4199741e36cf75256f (127.0.98.3:41535)
I20260924 04:41:10.024333 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57651
I20260924 04:41:10.024911 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:41:10.038230 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.029686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.0.98.62:40523)
0924 04:41:10.029821 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.029826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.030073 (+ 247us) server_negotiation.cc:407] Connection header received
0924 04:41:10.030334 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.030339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.030401 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.030501 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:10.031477 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.032052 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.032896 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.033132 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.033257 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.034762 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:10.034766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:10.034769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:10.034793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:10.036495 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.036919 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.036923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.036924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.037008 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.037300 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.037303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.037305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.037553 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:10.037679 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.037944 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.038053 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:10.045083 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:10.050971 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:10.068120 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.061524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42803 (local address 127.0.0.1:45454)
0924 04:41:10.062100 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:10.062151 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:10.062294 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:10.063398 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:10.063405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:10.063430 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:10.063586 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.063594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.065025 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.065028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.066087 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.066096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.067115 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.067461 (+ 346us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.067603 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.067662 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 04:41:10.067781 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:41:10.067921 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260924 04:41:10.068478 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.061736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38007 (local address 127.0.0.1:48424)
0924 04:41:10.062569 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:10.062584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:10.062663 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:10.063832 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:10.063835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:10.063855 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:10.063935 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.063941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.065477 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.065481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.066705 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.066714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.067808 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.067811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.067895 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.067928 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:41:10.067975 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:41:10.068053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":124,"threads_started":1}
I20260924 04:41:10.068480 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.061741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.98.3:41535)
0924 04:41:10.062189 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.062203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.062880 (+ 677us) server_negotiation.cc:407] Connection header received
0924 04:41:10.063019 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.063025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.063268 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.063444 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:10.063890 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.064978 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.066691 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.067518 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.067624 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.067771 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.068207 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.068324 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":127,"threads_started":1}
I20260924 04:41:10.068615 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.062408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.98.1:42803)
0924 04:41:10.062890 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.062895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.062914 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:41:10.062976 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.062979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.063223 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.063391 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:10.063717 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.064895 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.066215 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.066983 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.067096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.067280 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.068340 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.068444 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":128,"threads_started":1}
I20260924 04:41:10.069155 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.062370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.0.98.2:38007)
0924 04:41:10.063376 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.063387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.063409 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:41:10.063476 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.063481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.063638 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.064088 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:10.064244 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.065334 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.066828 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.067663 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.068769 (+ 1106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.068915 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.068969 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.069040 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":125,"threads_started":1}
I20260924 04:41:10.069514 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.061524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41535 (local address 127.0.0.1:37402)
0924 04:41:10.062780 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:10.062794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:10.062900 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:10.063595 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:10.063599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:10.063616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:10.063767 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.063778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.065156 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.065161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.066544 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.066555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.067835 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.067839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.067927 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.067949 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:41:10.068023 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:41:10.068085 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":791,"thread_start_us":106,"threads_started":1}
I20260924 04:41:10.071693 1087 tablet_service.cc:1467] Processing CreateTablet for tablet fc2982320e20425fa1b25d5ada6e4a4b (DEFAULT_TABLE table=test-table [id=5f95c3bace8f4240841390c9367c25c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:10.072216 1355 tablet_service.cc:1467] Processing CreateTablet for tablet fc2982320e20425fa1b25d5ada6e4a4b (DEFAULT_TABLE table=test-table [id=5f95c3bace8f4240841390c9367c25c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:10.072216 1221 tablet_service.cc:1467] Processing CreateTablet for tablet fc2982320e20425fa1b25d5ada6e4a4b (DEFAULT_TABLE table=test-table [id=5f95c3bace8f4240841390c9367c25c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:10.072692 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2982320e20425fa1b25d5ada6e4a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:10.073292 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2982320e20425fa1b25d5ada6e4a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:10.073292 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2982320e20425fa1b25d5ada6e4a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:10.079257 1438 tablet_bootstrap.cc:492] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: Bootstrap starting.
I20260924 04:41:10.080922 1439 tablet_bootstrap.cc:492] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: Bootstrap starting.
I20260924 04:41:10.082085 1438 tablet_bootstrap.cc:654] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:10.081938 1440 tablet_bootstrap.cc:492] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: Bootstrap starting.
I20260924 04:41:10.083077 1439 tablet_bootstrap.cc:654] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:10.083102 1438 log.cc:824] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:10.084350 1439 log.cc:824] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:10.084894 1440 tablet_bootstrap.cc:654] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:10.085456 1438 tablet_bootstrap.cc:492] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: No bootstrap required, opened a new log
I20260924 04:41:10.085650 1438 ts_tablet_manager.cc:1397] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 04:41:10.085778 1440 log.cc:824] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:10.086503 1439 tablet_bootstrap.cc:492] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: No bootstrap required, opened a new log
I20260924 04:41:10.086735 1439 ts_tablet_manager.cc:1397] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:41:10.087996 1440 tablet_bootstrap.cc:492] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: No bootstrap required, opened a new log
I20260924 04:41:10.088187 1440 ts_tablet_manager.cc:1397] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:41:10.089473 1438 raft_consensus.cc:348] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.089725 1438 raft_consensus.cc:374] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:10.089785 1438 raft_consensus.cc:729] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cafc7ddf8646c6994a2220a36e5954, State: Initialized, Role: FOLLOWER
I20260924 04:41:10.090248 1438 consensus_queue.cc:260] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.090896 1153 heartbeater.cc:503] Master 127.0.98.62:40523 was elected leader, sending a full tablet report...
I20260924 04:41:10.091312 1438 ts_tablet_manager.cc:1428] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:41:10.091404 1439 raft_consensus.cc:348] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.091688 1439 raft_consensus.cc:374] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:10.091749 1439 raft_consensus.cc:729] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4b193b80d04f4199741e36cf75256f, State: Initialized, Role: FOLLOWER
I20260924 04:41:10.092226 1439 consensus_queue.cc:260] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.092693 1440 raft_consensus.cc:348] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.092900 1421 heartbeater.cc:503] Master 127.0.98.62:40523 was elected leader, sending a full tablet report...
I20260924 04:41:10.092962 1440 raft_consensus.cc:374] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:10.093010 1440 raft_consensus.cc:729] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2ecbf125964fd9b7cd237d4f0c2bb2, State: Initialized, Role: FOLLOWER
I20260924 04:41:10.093290 1439 ts_tablet_manager.cc:1428] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:41:10.093472 1440 consensus_queue.cc:260] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.094478 1440 ts_tablet_manager.cc:1428] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:41:10.094870 1287 heartbeater.cc:503] Master 127.0.98.62:40523 was elected leader, sending a full tablet report...
W20260924 04:41:10.248488 1422 tablet.cc:2369] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:10.259902 1288 tablet.cc:2369] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:10.271445 1154 tablet.cc:2369] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:10.374253 1444 raft_consensus.cc:482] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:10.374507 1444 raft_consensus.cc:504] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.375917 1444 leader_election.cc:288] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c4b193b80d04f4199741e36cf75256f (127.0.98.3:41535), ab2ecbf125964fd9b7cd237d4f0c2bb2 (127.0.98.2:38007)
I20260924 04:41:10.381636 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.376342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41821 (local address 127.0.98.3:41535)
0924 04:41:10.376493 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.376498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.376841 (+ 343us) server_negotiation.cc:407] Connection header received
0924 04:41:10.377019 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.377022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.377069 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.377156 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:10.377605 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.378395 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.380301 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.380917 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.381037 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.381109 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.381417 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.381501 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 04:41:10.381636 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.376341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34043 (local address 127.0.98.2:38007)
0924 04:41:10.376493 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.376498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.376920 (+ 422us) server_negotiation.cc:407] Connection header received
0924 04:41:10.376996 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.376999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.377069 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.377273 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:10.377604 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.378448 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.380301 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.380917 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.381037 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.381110 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.381417 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.381500 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:10.381680 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.376241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41535 (local address 127.0.98.1:41821)
0924 04:41:10.376697 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:10.376715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:10.376875 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:10.377184 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:10.377187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:10.377292 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:10.377475 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.377482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.378575 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.378579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.380160 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.380171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.381080 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.381084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.381276 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.381293 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:41:10.381360 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:41:10.381481 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":144,"thread_start_us":132,"threads_started":1}
I20260924 04:41:10.381680 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.376241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38007 (local address 127.0.98.1:34043)
0924 04:41:10.376794 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:10.376807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:10.376890 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:10.377298 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:10.377300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:10.377312 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:10.377475 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.377482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.378575 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.378579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.380160 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:10.380171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.381080 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.381084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:10.381246 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.381273 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:41:10.381360 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:41:10.381481 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":222,"thread_start_us":92,"threads_started":1}
I20260924 04:41:10.382504 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2982320e20425fa1b25d5ada6e4a4b" candidate_uuid: "32cafc7ddf8646c6994a2220a36e5954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4b193b80d04f4199741e36cf75256f" is_pre_election: true
I20260924 04:41:10.382700 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2982320e20425fa1b25d5ada6e4a4b" candidate_uuid: "32cafc7ddf8646c6994a2220a36e5954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" is_pre_election: true
I20260924 04:41:10.382858 1375 raft_consensus.cc:2454] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32cafc7ddf8646c6994a2220a36e5954 in term 0.
I20260924 04:41:10.382998 1241 raft_consensus.cc:2454] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32cafc7ddf8646c6994a2220a36e5954 in term 0.
I20260924 04:41:10.383409 1042 leader_election.cc:302] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32cafc7ddf8646c6994a2220a36e5954, 4c4b193b80d04f4199741e36cf75256f; no voters:
I20260924 04:41:10.383739 1444 raft_consensus.cc:2781] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:10.383817 1444 raft_consensus.cc:482] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:10.383901 1444 raft_consensus.cc:3037] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:10.385047 1444 raft_consensus.cc:504] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.385411 1444 leader_election.cc:288] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [CANDIDATE]: Term 1 election: Requested vote from peers 4c4b193b80d04f4199741e36cf75256f (127.0.98.3:41535), ab2ecbf125964fd9b7cd237d4f0c2bb2 (127.0.98.2:38007)
I20260924 04:41:10.385788 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2982320e20425fa1b25d5ada6e4a4b" candidate_uuid: "32cafc7ddf8646c6994a2220a36e5954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4b193b80d04f4199741e36cf75256f"
I20260924 04:41:10.385788 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2982320e20425fa1b25d5ada6e4a4b" candidate_uuid: "32cafc7ddf8646c6994a2220a36e5954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2"
I20260924 04:41:10.385916 1241 raft_consensus.cc:3037] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:10.385916 1375 raft_consensus.cc:3037] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:10.387251 1241 raft_consensus.cc:2454] T fc2982320e20425fa1b25d5ada6e4a4b P ab2ecbf125964fd9b7cd237d4f0c2bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32cafc7ddf8646c6994a2220a36e5954 in term 1.
I20260924 04:41:10.387255 1375 raft_consensus.cc:2454] T fc2982320e20425fa1b25d5ada6e4a4b P 4c4b193b80d04f4199741e36cf75256f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32cafc7ddf8646c6994a2220a36e5954 in term 1.
I20260924 04:41:10.387547 1042 leader_election.cc:302] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32cafc7ddf8646c6994a2220a36e5954, 4c4b193b80d04f4199741e36cf75256f; no voters:
I20260924 04:41:10.387710 1444 raft_consensus.cc:2781] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:10.387933 1444 raft_consensus.cc:686] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [term 1 LEADER]: Becoming Leader. State: Replica: 32cafc7ddf8646c6994a2220a36e5954, State: Running, Role: LEADER
I20260924 04:41:10.388270 1444 consensus_queue.cc:237] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } }
I20260924 04:41:10.391453 962 catalog_manager.cc:5495] T fc2982320e20425fa1b25d5ada6e4a4b P 32cafc7ddf8646c6994a2220a36e5954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32cafc7ddf8646c6994a2220a36e5954 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "32cafc7ddf8646c6994a2220a36e5954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4b193b80d04f4199741e36cf75256f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32cafc7ddf8646c6994a2220a36e5954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab2ecbf125964fd9b7cd237d4f0c2bb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38007 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:10.439993 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.430887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.98.1:42803)
0924 04:41:10.431066 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.431070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.431342 (+ 272us) server_negotiation.cc:407] Connection header received
0924 04:41:10.431506 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.431510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.431561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.431695 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:10.432404 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.432920 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.433781 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.434000 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.434110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.436737 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:10.436748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:10.436785 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:10.436810 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:10.438384 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.438811 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.438815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.438817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.438906 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.439195 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.439197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.439198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.439440 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:10.439616 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.439743 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.439862 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:10.468245 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.458751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.98.1:42803)
0924 04:41:10.458934 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.458939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.459081 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:41:10.459355 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.459358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.459403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.459501 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:10.460195 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.460719 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.461626 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.461866 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.461984 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.464604 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:10.464608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:10.464611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:10.464643 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:10.466154 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.466790 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.466796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.466798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.466894 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.467217 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.467220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.467222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.467459 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:10.467623 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.467941 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.468077 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:10.494530 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:10.486167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.0.98.1:42803)
0924 04:41:10.486358 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:10.486364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:10.486602 (+ 238us) server_negotiation.cc:407] Connection header received
0924 04:41:10.486706 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:10.486709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:10.486761 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:10.486843 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:10.487603 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.488107 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.488963 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:10.489180 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:10.489275 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:10.491845 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:10.491848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:10.491850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:10.491874 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:10.493167 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.493556 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.493559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.493560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.493630 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:10.493853 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:10.493855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:10.493856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:10.494012 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:10.494109 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:10.494307 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 04:41:10.494407 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 04:41:10.496124 1067 tablet_service.cc:2908] Rejecting scan request for tablet fc2982320e20425fa1b25d5ada6e4a4b: Uninitialized: clean time has not yet been initialized
I20260924 04:41:10.498262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1024
I20260924 04:41:10.505007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1158
I20260924 04:41:10.510749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1292
I20260924 04:41:10.516642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 930
2026-09-24T04:41:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2704 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[1463](info): set up 2 sockets
Sep 24 04:41:10 dist-test-slave-npg2 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260924 04:41:11.575031 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:11 dist-test-slave-npg2 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:11.706643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:11.819370 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:11.819657 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:11.819697 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:11.823689 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:11.823745 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:41:11.823762 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:11.823781 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:11.823805 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:11.823820 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:11.829015 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:11.830039 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:11.831059 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:11.839792 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:11.839784 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:11.839781 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:11.839954 1476 server_base.cc:1034] running on GCE node
I20260924 04:41:11.840653 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:11.841818 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:11.842990 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790224871842975 us; error 28 us; skew 500 ppm
Sep 24 04:41:11 dist-test-slave-npg2 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:11.856583 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:11.857602 1476 webserver.cc:460] Webserver started at http://127.0.98.62:38129/ using document root <none> and password file <none>
I20260924 04:41:11.858237 1476 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:11.858337 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:11.858575 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:11.860811 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "f94efb51e1a845bd9d2a19a80be49008"
format_stamp: "Formatted at 2026-09-24 04:41:11 on dist-test-slave-npg2"
server_key: "618c64b83c71f691a6f81ab377db5ff3"
server_key_iv: "c9d76308929a2079710aeb15b6d5cf0e"
server_key_version: "encryptionkey@0"
I20260924 04:41:11.861382 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "f94efb51e1a845bd9d2a19a80be49008"
format_stamp: "Formatted at 2026-09-24 04:41:11 on dist-test-slave-npg2"
server_key: "618c64b83c71f691a6f81ab377db5ff3"
server_key_iv: "c9d76308929a2079710aeb15b6d5cf0e"
server_key_version: "encryptionkey@0"
I20260924 04:41:11.865270 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:11.867769 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:11.869247 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:11.869414 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "f94efb51e1a845bd9d2a19a80be49008"
format_stamp: "Formatted at 2026-09-24 04:41:11 on dist-test-slave-npg2"
server_key: "618c64b83c71f691a6f81ab377db5ff3"
server_key_iv: "c9d76308929a2079710aeb15b6d5cf0e"
server_key_version: "encryptionkey@0"
I20260924 04:41:11.869542 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:11.887596 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:11.891255 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:11.891475 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:11.900108 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41575
I20260924 04:41:11.900117 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41575 every 8 connection(s)
I20260924 04:41:11.901125 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:11.903740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1476
I20260924 04:41:11.903864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:11.904136 392 external_mini_cluster.cc:1428] Setting key 4ba64e92165bdcbb8cd230995df175d9
I20260924 04:41:11.904423 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:11.912076 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: Bootstrap starting.
Sep 24 04:41:11 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:11.914795 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:11.915618 1554 log.cc:824] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:11.917608 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: No bootstrap required, opened a new log
I20260924 04:41:11.919890 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:11.905588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.0.98.62:41575)
0924 04:41:11.905997 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:11.906019 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:11.906055 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:11.906724 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:11.906742 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:11.907083 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:11.907415 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:11.908353 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:11.909361 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:11.910545 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:11.911075 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:11.911341 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:11.914225 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:11.914231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:11.914246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:11.914277 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:11.916789 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:11.917302 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:11.917308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:11.917321 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:11.917452 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:11.917769 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:11.917772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:11.917774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:11.918225 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:11.918447 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:11.918761 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 04:41:11.919072 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":144,"threads_started":1}
I20260924 04:41:11.921038 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94efb51e1a845bd9d2a19a80be49008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41575 } }
I20260924 04:41:11.921325 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:11.921394 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94efb51e1a845bd9d2a19a80be49008, State: Initialized, Role: FOLLOWER
I20260924 04:41:11.921854 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94efb51e1a845bd9d2a19a80be49008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41575 } }
I20260924 04:41:11.922042 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:11.922139 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:11.922266 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:11.923316 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94efb51e1a845bd9d2a19a80be49008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41575 } }
I20260924 04:41:11.923666 1554 leader_election.cc:302] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f94efb51e1a845bd9d2a19a80be49008; no voters:
I20260924 04:41:11.923985 1554 leader_election.cc:288] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:11.924120 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:11.924364 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [term 1 LEADER]: Becoming Leader. State: Replica: f94efb51e1a845bd9d2a19a80be49008, State: Running, Role: LEADER
I20260924 04:41:11.924747 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94efb51e1a845bd9d2a19a80be49008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41575 } }
I20260924 04:41:11.925308 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:11.926496 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f94efb51e1a845bd9d2a19a80be49008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94efb51e1a845bd9d2a19a80be49008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41575 } } }
I20260924 04:41:11.926620 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:11.926748 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f94efb51e1a845bd9d2a19a80be49008. Latest consensus state: current_term: 1 leader_uuid: "f94efb51e1a845bd9d2a19a80be49008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94efb51e1a845bd9d2a19a80be49008" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41575 } } }
I20260924 04:41:11.926827 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:11.927330 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:11.930135 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:11.935648 1568 catalog_manager.cc:1347] Generated new cluster ID: dcd80714f6fc4c50af95f4b1d05a0c7e
I20260924 04:41:11.935729 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:11.947793 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:11.948802 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:11.955256 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: Generated new TSK 0
I20260924 04:41:11.956130 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:11.991672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41575
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:12.107616 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:12.107894 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:12.107940 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:12.112023 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:12.112095 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:12.112134 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:12.112222 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:12.117524 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:12.118678 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:12.119783 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:12.129751 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:12.129711 1580 server_base.cc:1034] running on GCE node
W20260924 04:41:12.129695 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:12.129848 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:12.130398 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:12.131111 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:12.132432 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790224872132417 us; error 34 us; skew 500 ppm
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:12.135362 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:12.136256 1580 webserver.cc:460] Webserver started at http://127.0.98.1:32847/ using document root <none> and password file <none>
I20260924 04:41:12.136790 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:12.136849 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:12.137014 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:12.139279 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "84759bf4644443c3a78d7f65b8804e19"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "def4f776983ea938f718258631830047"
server_key_iv: "4fdd19c20a33dd379aa1669a714002a1"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.139829 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84759bf4644443c3a78d7f65b8804e19"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "def4f776983ea938f718258631830047"
server_key_iv: "4fdd19c20a33dd379aa1669a714002a1"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.143981 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:41:12.146528 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.147933 1580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:12.148079 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "84759bf4644443c3a78d7f65b8804e19"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "def4f776983ea938f718258631830047"
server_key_iv: "4fdd19c20a33dd379aa1669a714002a1"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.148206 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:12.164963 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:12.168534 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:12.168768 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:12.170269 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:12.170347 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.170418 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:12.170468 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.181622 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44851
I20260924 04:41:12.181665 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44851 every 8 connection(s)
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:12.182679 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:12.186168 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.171477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58403 (local address 127.0.98.62:41575)
0924 04:41:12.171615 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.171618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.172742 (+ 1124us) server_negotiation.cc:407] Connection header received
0924 04:41:12.174103 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.174108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.174163 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.174216 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.176063 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.176995 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.178830 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.179060 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.179247 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.182391 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.182394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.182396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.182422 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.184211 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.184763 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.184766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.184767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.184855 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.185163 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.185165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.185166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.185401 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.185482 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.185935 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.186045 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:41:12.187036 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.172040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.98.1:58403)
0924 04:41:12.172558 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.172648 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.173831 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.174616 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.174623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.175169 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:12.175894 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.175917 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.177149 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.177154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.178711 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.178723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.179296 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.179303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.179535 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.180289 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:12.180313 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:12.182199 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:12.184395 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.184401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.184412 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.184670 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.184940 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.184943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.184944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.185076 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.185495 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:12.185500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:12.185730 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.185890 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.186148 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
I20260924 04:41:12.188552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1580
I20260924 04:41:12.188705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:12.188984 392 external_mini_cluster.cc:1428] Setting key f4dedd5cb2148312dd320fac1ba92a6d
I20260924 04:41:12.189390 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41575
I20260924 04:41:12.189553 1711 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:12.190124 1711 heartbeater.cc:511] Master 127.0.98.62:41575 requested a full tablet report, sending...
I20260924 04:41:12.191921 1506 ts_manager.cc:194] Registered new tserver with Master: 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1:44851)
I20260924 04:41:12.193570 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58403
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:12.199661 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.191099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33673 (local address 127.0.98.62:41575)
0924 04:41:12.191269 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.191275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.191563 (+ 288us) server_negotiation.cc:407] Connection header received
0924 04:41:12.191677 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.191680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.191724 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.191807 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.192432 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.192981 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.193990 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.194340 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.194446 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.195978 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.195982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.195985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.196011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.197761 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.198302 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.198305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.198307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.198388 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.198729 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.198732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.198733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.198939 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.199026 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.199412 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.199544 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:41:12.199733 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.191011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.98.1:33673)
0924 04:41:12.191450 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.191467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.191579 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.191859 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.191865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.192114 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:12.192277 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.192288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.193112 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.193114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.193864 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.193877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.194468 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.194475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.194604 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.195242 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:12.195271 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:12.195771 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:12.197882 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.197888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.197890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.198187 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.198493 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.198496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.198498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.198601 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.199048 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:12.199051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:12.199178 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.199304 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.199478 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:12.219408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41575
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:12.334839 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:12.335179 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:12.335227 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:12.339335 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:12.339406 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:12.339447 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:12.339488 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:12.344956 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:12.346108 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:12.347189 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:12.357415 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:12.357470 1720 server_base.cc:1034] running on GCE node
W20260924 04:41:12.357455 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:12.357415 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:12.358239 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:12.358911 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:12.360088 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790224872360094 us; error 58 us; skew 500 ppm
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:12.363050 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:12.363979 1720 webserver.cc:460] Webserver started at http://127.0.98.2:42371/ using document root <none> and password file <none>
I20260924 04:41:12.364630 1720 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:12.364696 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:12.364862 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:12.366942 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "5afd6625b4d94968957685aeb518647b"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "2cc62ebd8092bb852e66b4c21ee613bf"
server_key_iv: "b0a3ba9f4c16e4f2c1266454607dd800"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.367624 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5afd6625b4d94968957685aeb518647b"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "2cc62ebd8092bb852e66b4c21ee613bf"
server_key_iv: "b0a3ba9f4c16e4f2c1266454607dd800"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.371618 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:12.374204 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.375576 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:12.375722 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "5afd6625b4d94968957685aeb518647b"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "2cc62ebd8092bb852e66b4c21ee613bf"
server_key_iv: "b0a3ba9f4c16e4f2c1266454607dd800"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.375859 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:12.392645 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:12.396214 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:12.396446 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:12.397903 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:12.397986 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.398059 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:12.398093 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:12.409471 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46389
I20260924 04:41:12.409489 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46389 every 8 connection(s)
I20260924 04:41:12.410625 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:12.412963 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.398975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38217 (local address 127.0.98.62:41575)
0924 04:41:12.399102 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.399106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.400239 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 04:41:12.401411 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.401415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.401509 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.401615 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.402914 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.403583 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.405294 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.405720 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.405841 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.408643 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.408647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.408650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.408692 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.410608 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.411380 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.411384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.411386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.411488 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.411839 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.411843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.411844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.412039 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.412153 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.412649 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.412779 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:41:12.413838 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.399580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.98.2:38217)
0924 04:41:12.400075 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.400118 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.401102 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.401812 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.401818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.402149 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:12.402740 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.402776 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.403727 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.403730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.405097 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.405106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.405898 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.405902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.406118 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.406755 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:12.406773 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:12.408430 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:12.410733 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.410738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.410749 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.411095 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.411573 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.411578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.411580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.411722 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.412148 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:12.412155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:12.412402 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.412593 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.412924 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260924 04:41:12.415531 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41575
I20260924 04:41:12.415704 1851 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:12.416368 1851 heartbeater.cc:511] Master 127.0.98.62:41575 requested a full tablet report, sending...
I20260924 04:41:12.416504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1720
I20260924 04:41:12.416625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:12.416877 392 external_mini_cluster.cc:1428] Setting key 06ec0497aab891af044c9ee834cc3995
I20260924 04:41:12.417550 1506 ts_manager.cc:194] Registered new tserver with Master: 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389)
I20260924 04:41:12.418392 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38217
I20260924 04:41:12.427074 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.417876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48677 (local address 127.0.98.62:41575)
0924 04:41:12.417987 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.417992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.418257 (+ 265us) server_negotiation.cc:407] Connection header received
0924 04:41:12.418376 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.418379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.418425 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.418511 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.419038 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.419772 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.421052 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.421264 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.421356 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.422987 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.422991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.422994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.423022 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.424860 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.425471 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.425475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.425477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.425607 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.425922 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.425927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.425929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.426249 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.426358 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.426775 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.426896 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:41:12.427443 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.417802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.98.2:48677)
0924 04:41:12.418162 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.418177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.418281 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.418552 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.418555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.418789 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:12.418893 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.418899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.419952 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.419957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.420914 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.420926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.421389 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.421393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.421601 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.422201 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:12.422221 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:12.422806 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:12.424987 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.424990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.424992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.425298 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.425723 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.425727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.425730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.425823 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.426452 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:12.426458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:12.426580 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.426709 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.426929 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":56,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:12.450470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41575
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:12.568192 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:12.568507 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:12.568614 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:12.573323 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:12.573417 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:12.573490 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:12.573582 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:12.579910 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:12.581174 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:12.582393 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:12.593225 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:12.593279 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:12.593400 1860 server_base.cc:1034] running on GCE node
W20260924 04:41:12.593247 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:12.594019 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:12.594645 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:12.595844 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790224872595825 us; error 33 us; skew 500 ppm
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:12.598677 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:12.599622 1860 webserver.cc:460] Webserver started at http://127.0.98.3:45953/ using document root <none> and password file <none>
I20260924 04:41:12.600191 1860 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:12.600283 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:12.600565 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:12.602622 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "41f73a1a7a91484da32614d363918564"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "1447286d0fd85e26df35d9e8646f39b0"
server_key_iv: "ee0650f4f4727e41836ec62c73ead12a"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.603279 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "41f73a1a7a91484da32614d363918564"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "1447286d0fd85e26df35d9e8646f39b0"
server_key_iv: "ee0650f4f4727e41836ec62c73ead12a"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.607060 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:12.609625 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.610934 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:41:12.611075 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "41f73a1a7a91484da32614d363918564"
format_stamp: "Formatted at 2026-09-24 04:41:12 on dist-test-slave-npg2"
server_key: "1447286d0fd85e26df35d9e8646f39b0"
server_key_iv: "ee0650f4f4727e41836ec62c73ead12a"
server_key_version: "encryptionkey@0"
I20260924 04:41:12.611260 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:12.654701 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:12.658023 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:12.658262 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:12.659929 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:12.660006 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:12.660077 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:12.660126 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:12.671455 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44203
I20260924 04:41:12.671509 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44203 every 8 connection(s)
I20260924 04:41:12.672713 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:12.674573 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.660953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39161 (local address 127.0.98.62:41575)
0924 04:41:12.661084 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.661087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.662204 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 04:41:12.663388 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.663392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.663463 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.663574 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.664915 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.665478 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.666788 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.667028 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.667190 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.670479 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.670482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.670484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.670510 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.672319 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.672840 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.672844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.672845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.672939 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.673365 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.673368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.673370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.673628 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.673776 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.674345 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.674453 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:12.675462 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.661562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.98.3:39161)
0924 04:41:12.662044 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.662088 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.663072 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.663895 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.663901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.664239 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:12.664736 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.664750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.665616 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.665620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.666653 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.666662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.667233 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.667237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.667455 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.668247 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:12.668263 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:12.670248 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:12.672445 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.672451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.672468 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.672718 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.673029 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.673032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.673035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.673244 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.673764 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:12.673769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:12.674087 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.674279 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.674544 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260924 04:41:12.676723 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41575
I20260924 04:41:12.676908 1991 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:12.677456 1991 heartbeater.cc:511] Master 127.0.98.62:41575 requested a full tablet report, sending...
I20260924 04:41:12.678390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 1860
I20260924 04:41:12.678507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:12.678637 1508 ts_manager.cc:194] Registered new tserver with Master: 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203)
I20260924 04:41:12.678808 392 external_mini_cluster.cc:1428] Setting key 3e6d024725f2740cf51ff3c24e45139a
I20260924 04:41:12.679621 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39161
I20260924 04:41:12.681080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:41:12.689107 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.679204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56273 (local address 127.0.98.62:41575)
0924 04:41:12.679330 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.679334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.680861 (+ 1527us) server_negotiation.cc:407] Connection header received
0924 04:41:12.681011 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.681014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.681064 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.681147 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.681702 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.682525 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.683841 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.684052 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.684166 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.685449 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.685453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.685456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.685479 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.687298 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.687769 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.687774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.687776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.687890 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.688234 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.688238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.688240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.688419 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.688497 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.688893 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.688989 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:41:12.689287 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.679084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.98.3:56273)
0924 04:41:12.680144 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.680789 (+ 645us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.680905 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.681188 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.681191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.681446 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:12.681570 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.681578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.682658 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.682661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.683709 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.683725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.684178 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.684182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.684368 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.684927 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:12.684939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:12.685274 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:12.687416 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.687419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.687420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.687630 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.687999 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:12.688004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:12.688007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:12.688115 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:12.688518 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:12.688523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:12.688695 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.688827 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.689010 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":132,"threads_started":1}
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:41:12 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:12.710729 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.701871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.0.98.62:41575)
0924 04:41:12.701996 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.702000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.702012 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:12.702050 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.702053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.702090 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.702187 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:12.702968 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.703619 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.704481 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.704708 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.704796 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.707444 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:12.707447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:12.707449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:12.707475 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:12.709001 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.709496 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.709499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.709500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.709577 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:12.709887 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:12.709891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:12.709892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:12.710084 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:12.710200 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.710486 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.710595 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:12.713763 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:12.717147 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:12.732120 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.726229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.0.98.1:44851)
0924 04:41:12.727588 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.727595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.727618 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:41:12.727692 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.727697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.727897 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.728070 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:12.728337 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.729088 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.730513 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.731394 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.731503 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.731645 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.731877 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.731975 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":92,"threads_started":1}
I20260924 04:41:12.732139 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.726024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44851 (local address 127.0.0.1:60852)
0924 04:41:12.726736 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.726786 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.726932 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.728059 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.728062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.728078 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:12.728208 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.728217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.729205 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.729211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.730378 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.730389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.731533 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.731537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.731666 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.731710 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.731836 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.731962 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":27,"spinlock_wait_cycles":5888,"thread_start_us":96,"threads_started":1}
I20260924 04:41:12.732672 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.726160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44203 (local address 127.0.0.1:44046)
0924 04:41:12.726736 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.726790 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.726934 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.727908 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.727914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.727940 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:12.728145 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.728152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.729876 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.729880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.731069 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.731079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.732247 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.732253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.732350 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.732374 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.732423 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.732495 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":62,"spinlock_wait_cycles":4992,"thread_start_us":135,"threads_started":1}
I20260924 04:41:12.733142 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.726774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.0.1:51176)
0924 04:41:12.727261 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.727273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.727361 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.728187 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.728190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.728205 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:12.728341 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.728347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.729912 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.729915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.731190 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.731202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.732492 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.732498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.732863 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.732886 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.732963 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.733026 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":33,"thread_start_us":102,"threads_started":1}
I20260924 04:41:12.733283 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.727258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.98.2:46389)
0924 04:41:12.727671 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.727681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.727698 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:41:12.727758 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.727762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.727899 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.728049 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:12.728475 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.729732 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.731326 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.732123 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.732245 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.732414 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.733024 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.733139 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":120,"threads_started":1}
I20260924 04:41:12.733536 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.726854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.0.98.3:44203)
0924 04:41:12.727291 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.727362 (+ 71us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.727392 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:41:12.727480 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.727487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.727716 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.728316 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:12.728467 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.729748 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.731223 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.732095 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.732900 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.733078 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.733127 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.733212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260924 04:41:12.736143 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 6c076993fbad40a4aa0611d1efc851b8 (DEFAULT_TABLE table=test-table [id=76a9e59649224e5b96e60f76efd50f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:12.736130 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 6c076993fbad40a4aa0611d1efc851b8 (DEFAULT_TABLE table=test-table [id=76a9e59649224e5b96e60f76efd50f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:12.736132 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 6c076993fbad40a4aa0611d1efc851b8 (DEFAULT_TABLE table=test-table [id=76a9e59649224e5b96e60f76efd50f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:12.737213 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c076993fbad40a4aa0611d1efc851b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:12.737215 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c076993fbad40a4aa0611d1efc851b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:12.737217 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c076993fbad40a4aa0611d1efc851b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:12.745304 2014 tablet_bootstrap.cc:492] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Bootstrap starting.
I20260924 04:41:12.745703 2015 tablet_bootstrap.cc:492] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Bootstrap starting.
I20260924 04:41:12.747502 2016 tablet_bootstrap.cc:492] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Bootstrap starting.
I20260924 04:41:12.747619 2015 tablet_bootstrap.cc:654] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:12.747610 2014 tablet_bootstrap.cc:654] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:12.748404 2014 log.cc:824] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:12.748517 2015 log.cc:824] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:12.749497 2016 tablet_bootstrap.cc:654] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:12.750285 2016 log.cc:824] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:12.750571 2014 tablet_bootstrap.cc:492] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: No bootstrap required, opened a new log
I20260924 04:41:12.750770 2014 ts_tablet_manager.cc:1397] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:41:12.752287 2015 tablet_bootstrap.cc:492] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: No bootstrap required, opened a new log
I20260924 04:41:12.752501 2016 tablet_bootstrap.cc:492] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: No bootstrap required, opened a new log
I20260924 04:41:12.752533 2015 ts_tablet_manager.cc:1397] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:41:12.752676 2016 ts_tablet_manager.cc:1397] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260924 04:41:12.755330 2014 raft_consensus.cc:348] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.755637 2014 raft_consensus.cc:374] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:12.755688 2014 raft_consensus.cc:729] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afd6625b4d94968957685aeb518647b, State: Initialized, Role: FOLLOWER
I20260924 04:41:12.755785 2015 raft_consensus.cc:348] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.756016 2015 raft_consensus.cc:374] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:12.756062 2015 raft_consensus.cc:729] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84759bf4644443c3a78d7f65b8804e19, State: Initialized, Role: FOLLOWER
I20260924 04:41:12.756203 2014 consensus_queue.cc:260] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.756410 2015 consensus_queue.cc:260] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.756982 1851 heartbeater.cc:503] Master 127.0.98.62:41575 was elected leader, sending a full tablet report...
I20260924 04:41:12.756935 2016 raft_consensus.cc:348] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.757171 2016 raft_consensus.cc:374] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:12.757221 2016 raft_consensus.cc:729] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f73a1a7a91484da32614d363918564, State: Initialized, Role: FOLLOWER
I20260924 04:41:12.757328 2015 ts_tablet_manager.cc:1428] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:41:12.757339 2014 ts_tablet_manager.cc:1428] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:41:12.757625 1711 heartbeater.cc:503] Master 127.0.98.62:41575 was elected leader, sending a full tablet report...
I20260924 04:41:12.757643 2016 consensus_queue.cc:260] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.758627 2016 ts_tablet_manager.cc:1428] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260924 04:41:12.758948 1991 heartbeater.cc:503] Master 127.0.98.62:41575 was elected leader, sending a full tablet report...
W20260924 04:41:12.911868 1852 tablet.cc:2369] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:12.923877 1992 tablet.cc:2369] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:12.934516 1712 tablet.cc:2369] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:12.970245 2021 raft_consensus.cc:482] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:12.970526 2021 raft_consensus.cc:504] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.971772 2021 leader_election.cc:288] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203), 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389)
I20260924 04:41:12.976161 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.972229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45485 (local address 127.0.98.3:44203)
0924 04:41:12.972359 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.972364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.972632 (+ 268us) server_negotiation.cc:407] Connection header received
0924 04:41:12.972832 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.972838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.972890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.972986 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:12.973264 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.973922 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.975073 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.975537 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.975620 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.975674 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.975946 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.976028 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:41:12.976249 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.972192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57777 (local address 127.0.98.2:46389)
0924 04:41:12.972374 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:12.972381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:12.972718 (+ 337us) server_negotiation.cc:407] Connection header received
0924 04:41:12.972811 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:12.972814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:12.972874 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:12.972935 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:12.973392 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.974053 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.975228 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.975660 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.975763 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.975825 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:12.976035 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 04:41:12.976133 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:41:12.976291 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.972085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44203 (local address 127.0.98.1:45485)
0924 04:41:12.972524 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.972538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.972714 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.973015 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.973018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.973042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:12.973174 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.973179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.974025 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.974028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.974943 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.974951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.975660 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.975664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.975770 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.975793 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.975917 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.976018 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":19,"spinlock_wait_cycles":48128,"thread_start_us":102,"threads_started":1}
I20260924 04:41:12.976580 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:12.972085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.1:57777)
0924 04:41:12.972527 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:12.972539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:12.972712 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:12.972974 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:12.972978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:12.972994 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:12.973211 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.973219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.974211 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.974215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.975094 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:12.975104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:12.975787 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:12.975789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:12.975873 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:12.975921 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 04:41:12.976001 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:41:12.976061 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":213,"spinlock_wait_cycles":5632,"thread_start_us":84,"threads_started":1}
I20260924 04:41:12.977036 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c076993fbad40a4aa0611d1efc851b8" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f73a1a7a91484da32614d363918564" is_pre_election: true
I20260924 04:41:12.977274 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c076993fbad40a4aa0611d1efc851b8" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afd6625b4d94968957685aeb518647b" is_pre_election: true
I20260924 04:41:12.977375 1945 raft_consensus.cc:2454] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 0.
I20260924 04:41:12.977609 1805 raft_consensus.cc:2454] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 0.
I20260924 04:41:12.977948 1598 leader_election.cc:302] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f73a1a7a91484da32614d363918564, 84759bf4644443c3a78d7f65b8804e19; no voters:
I20260924 04:41:12.978250 2021 raft_consensus.cc:2781] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:12.978355 2021 raft_consensus.cc:482] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:12.978417 2021 raft_consensus.cc:3037] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:12.979547 2021 raft_consensus.cc:504] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.979902 2021 leader_election.cc:288] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 election: Requested vote from peers 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203), 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389)
I20260924 04:41:12.980257 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c076993fbad40a4aa0611d1efc851b8" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f73a1a7a91484da32614d363918564"
I20260924 04:41:12.980279 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c076993fbad40a4aa0611d1efc851b8" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afd6625b4d94968957685aeb518647b"
I20260924 04:41:12.980393 1805 raft_consensus.cc:3037] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:12.980405 1945 raft_consensus.cc:3037] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:12.981732 1805 raft_consensus.cc:2454] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 1.
I20260924 04:41:12.982036 1945 raft_consensus.cc:2454] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 1.
I20260924 04:41:12.982066 1596 leader_election.cc:302] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5afd6625b4d94968957685aeb518647b, 84759bf4644443c3a78d7f65b8804e19; no voters:
I20260924 04:41:12.982232 2021 raft_consensus.cc:2781] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:12.982462 2021 raft_consensus.cc:686] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 1 LEADER]: Becoming Leader. State: Replica: 84759bf4644443c3a78d7f65b8804e19, State: Running, Role: LEADER
I20260924 04:41:12.982856 2021 consensus_queue.cc:237] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } }
I20260924 04:41:12.987013 1507 catalog_manager.cc:5495] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "84759bf4644443c3a78d7f65b8804e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:13.037750 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.033027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.0.98.1:44851)
0924 04:41:13.033181 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.033187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.033296 (+ 109us) server_negotiation.cc:407] Connection header received
0924 04:41:13.033496 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.033500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.033582 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.033761 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:13.033994 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.034866 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.035826 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.036220 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.036347 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.036639 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.037212 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.037345 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.037495 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.037568 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:41:13.044507 1945 raft_consensus.cc:1261] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Refusing update from remote peer 84759bf4644443c3a78d7f65b8804e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:13.044507 1805 raft_consensus.cc:1261] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Refusing update from remote peer 84759bf4644443c3a78d7f65b8804e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:13.045262 2021 consensus_queue.cc:1035] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:13.045475 2025 consensus_queue.cc:1035] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:13.057642 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332761079981817856 to 7332761079741272064. Current Snapshot: MvccSnapshot[applied={T|T < 7332761079981817856}]
I20260924 04:41:13.064498 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:13.064653 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:13.067200 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: Sending DeleteTablet for 3 replicas of tablet 6c076993fbad40a4aa0611d1efc851b8
I20260924 04:41:13.067994 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c076993fbad40a4aa0611d1efc851b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44046
I20260924 04:41:13.068171 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c076993fbad40a4aa0611d1efc851b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60852
I20260924 04:41:13.068167 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c076993fbad40a4aa0611d1efc851b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51176
I20260924 04:41:13.068555 2037 tablet_replica.cc:330] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: stopping tablet replica
I20260924 04:41:13.068907 2038 tablet_replica.cc:330] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: stopping tablet replica
I20260924 04:41:13.068889 2037 raft_consensus.cc:2229] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:13.069145 2038 raft_consensus.cc:2229] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:13.069211 2037 raft_consensus.cc:2258] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:13.069573 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:13.069998 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:13.070012 2039 tablet_replica.cc:330] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: stopping tablet replica
I20260924 04:41:13.070292 2039 raft_consensus.cc:2229] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:13.070585 2039 raft_consensus.cc:2258] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:13.070876 2038 raft_consensus.cc:2258] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:13.071321 2037 ts_tablet_manager.cc:1905] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:13.072259 2038 ts_tablet_manager.cc:1905] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:13.074186 2037 ts_tablet_manager.cc:1918] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:13.074287 2037 log.cc:1196] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/6c076993fbad40a4aa0611d1efc851b8
I20260924 04:41:13.074615 2037 ts_tablet_manager.cc:1939] T 6c076993fbad40a4aa0611d1efc851b8 P 41f73a1a7a91484da32614d363918564: Deleting consensus metadata
I20260924 04:41:13.075521 2038 ts_tablet_manager.cc:1918] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:13.075614 2038 log.cc:1196] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/6c076993fbad40a4aa0611d1efc851b8
I20260924 04:41:13.076072 2038 ts_tablet_manager.cc:1939] T 6c076993fbad40a4aa0611d1efc851b8 P 84759bf4644443c3a78d7f65b8804e19: Deleting consensus metadata
I20260924 04:41:13.076200 1494 catalog_manager.cc:4841] TS 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203): tablet 6c076993fbad40a4aa0611d1efc851b8 (table test-table [id=76a9e59649224e5b96e60f76efd50f07]) successfully deleted
I20260924 04:41:13.076447 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd8ce135615454cb51c59d6dedb1cd2 (DEFAULT_TABLE table=test-table [id=eecc77bc6a4046eebd763257bfe8eb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:13.076723 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8ce135615454cb51c59d6dedb1cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:13.076690 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd8ce135615454cb51c59d6dedb1cd2 (DEFAULT_TABLE table=test-table [id=eecc77bc6a4046eebd763257bfe8eb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:13.076928 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8ce135615454cb51c59d6dedb1cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:13.078866 2039 ts_tablet_manager.cc:1905] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:13.079236 2016 tablet_bootstrap.cc:492] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Bootstrap starting.
I20260924 04:41:13.079464 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd8ce135615454cb51c59d6dedb1cd2 (DEFAULT_TABLE table=test-table [id=eecc77bc6a4046eebd763257bfe8eb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:13.079722 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8ce135615454cb51c59d6dedb1cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:13.080214 2016 tablet_bootstrap.cc:654] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:13.081030 1493 catalog_manager.cc:4841] TS 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1:44851): tablet 6c076993fbad40a4aa0611d1efc851b8 (table test-table [id=76a9e59649224e5b96e60f76efd50f07]) successfully deleted
I20260924 04:41:13.081436 2039 ts_tablet_manager.cc:1918] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:13.081557 2039 log.cc:1196] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/6c076993fbad40a4aa0611d1efc851b8
I20260924 04:41:13.081885 2039 ts_tablet_manager.cc:1939] T 6c076993fbad40a4aa0611d1efc851b8 P 5afd6625b4d94968957685aeb518647b: Deleting consensus metadata
I20260924 04:41:13.082240 2015 tablet_bootstrap.cc:492] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Bootstrap starting.
I20260924 04:41:13.083222 2015 tablet_bootstrap.cc:654] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:13.083349 2016 tablet_bootstrap.cc:492] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: No bootstrap required, opened a new log
I20260924 04:41:13.083420 2016 ts_tablet_manager.cc:1397] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 04:41:13.083755 2014 tablet_bootstrap.cc:492] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Bootstrap starting.
I20260924 04:41:13.083873 2016 raft_consensus.cc:348] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.083981 2016 raft_consensus.cc:374] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:13.084010 2016 raft_consensus.cc:729] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f73a1a7a91484da32614d363918564, State: Initialized, Role: FOLLOWER
I20260924 04:41:13.084123 2016 consensus_queue.cc:260] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.084352 2016 ts_tablet_manager.cc:1428] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:13.084429 2015 tablet_bootstrap.cc:492] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: No bootstrap required, opened a new log
I20260924 04:41:13.084497 2015 ts_tablet_manager.cc:1397] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 04:41:13.084520 1492 catalog_manager.cc:4841] TS 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389): tablet 6c076993fbad40a4aa0611d1efc851b8 (table test-table [id=76a9e59649224e5b96e60f76efd50f07]) successfully deleted
I20260924 04:41:13.084718 2014 tablet_bootstrap.cc:654] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:13.085011 2015 raft_consensus.cc:348] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.085124 2015 raft_consensus.cc:374] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:13.085161 2015 raft_consensus.cc:729] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84759bf4644443c3a78d7f65b8804e19, State: Initialized, Role: FOLLOWER
I20260924 04:41:13.085294 2015 consensus_queue.cc:260] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.085541 2015 ts_tablet_manager.cc:1428] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:41:13.086328 2014 tablet_bootstrap.cc:492] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: No bootstrap required, opened a new log
I20260924 04:41:13.086407 2014 ts_tablet_manager.cc:1397] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:13.086834 2014 raft_consensus.cc:348] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.086961 2014 raft_consensus.cc:374] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:13.086995 2014 raft_consensus.cc:729] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afd6625b4d94968957685aeb518647b, State: Initialized, Role: FOLLOWER
I20260924 04:41:13.087157 2014 consensus_queue.cc:260] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.087425 2014 ts_tablet_manager.cc:1428] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:13.136075 2021 raft_consensus.cc:482] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:13.136245 2021 raft_consensus.cc:504] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.136759 2021 leader_election.cc:288] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389), 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203)
I20260924 04:41:13.137199 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8ce135615454cb51c59d6dedb1cd2" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f73a1a7a91484da32614d363918564" is_pre_election: true
I20260924 04:41:13.137212 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8ce135615454cb51c59d6dedb1cd2" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afd6625b4d94968957685aeb518647b" is_pre_election: true
I20260924 04:41:13.137367 1945 raft_consensus.cc:2454] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 0.
I20260924 04:41:13.137373 1805 raft_consensus.cc:2454] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 0.
I20260924 04:41:13.137651 1598 leader_election.cc:302] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f73a1a7a91484da32614d363918564, 84759bf4644443c3a78d7f65b8804e19; no voters:
I20260924 04:41:13.137856 2021 raft_consensus.cc:2781] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:13.137943 2021 raft_consensus.cc:482] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:13.138000 2021 raft_consensus.cc:3037] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:13.139016 2021 raft_consensus.cc:504] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.139439 2021 leader_election.cc:288] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 election: Requested vote from peers 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389), 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203)
I20260924 04:41:13.139814 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8ce135615454cb51c59d6dedb1cd2" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f73a1a7a91484da32614d363918564"
I20260924 04:41:13.139819 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8ce135615454cb51c59d6dedb1cd2" candidate_uuid: "84759bf4644443c3a78d7f65b8804e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afd6625b4d94968957685aeb518647b"
I20260924 04:41:13.139946 1805 raft_consensus.cc:3037] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:13.139950 1945 raft_consensus.cc:3037] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:13.140921 1945 raft_consensus.cc:2454] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 1.
I20260924 04:41:13.141114 1805 raft_consensus.cc:2454] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84759bf4644443c3a78d7f65b8804e19 in term 1.
I20260924 04:41:13.141254 1598 leader_election.cc:302] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f73a1a7a91484da32614d363918564, 84759bf4644443c3a78d7f65b8804e19; no voters:
I20260924 04:41:13.141434 2021 raft_consensus.cc:2781] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:13.141520 2021 raft_consensus.cc:686] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 1 LEADER]: Becoming Leader. State: Replica: 84759bf4644443c3a78d7f65b8804e19, State: Running, Role: LEADER
I20260924 04:41:13.141659 2021 consensus_queue.cc:237] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.143445 1507 catalog_manager.cc:5495] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "84759bf4644443c3a78d7f65b8804e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:41:13 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:13.178609 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.168882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41575 (local address 127.0.0.1:45098)
0924 04:41:13.169184 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:13.169201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:13.169329 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:13.169605 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:13.169607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:13.169975 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:13.170064 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.170069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.171112 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.171114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.171729 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.171738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.172355 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.172357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.172419 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.173024 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:13.173036 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:13.174711 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:13.176723 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:13.176731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:13.176735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:13.177037 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:13.177383 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:13.177386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:13.177387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:13.177477 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:13.177993 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:13.178004 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:13.178120 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 04:41:13.178314 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 04:41:13.178463 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260924 04:41:13.178738 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.168998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.98.62:41575)
0924 04:41:13.169147 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.169152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.169294 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:41:13.169419 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.169422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.169474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.169578 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:13.170182 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.170996 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.171859 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.172192 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.172303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.174873 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:13.174877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:13.174882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:13.174909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:13.176617 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:13.177137 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:13.177141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:13.177143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:13.177233 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:13.177586 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:13.177590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:13.177592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:13.177829 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:13.177981 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.178390 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.178543 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":54}
I20260924 04:41:13.183429 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45098:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:41:13.184273 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:13.189522 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 43ab4251fd984128b2c294dc9c3b1a18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=398ed2a8fd3e4574924c924eca2fe793]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:13.189908 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ab4251fd984128b2c294dc9c3b1a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:13.189852 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 43ab4251fd984128b2c294dc9c3b1a18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=398ed2a8fd3e4574924c924eca2fe793]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:13.190181 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ab4251fd984128b2c294dc9c3b1a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:13.190231 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 43ab4251fd984128b2c294dc9c3b1a18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=398ed2a8fd3e4574924c924eca2fe793]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:13.190590 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ab4251fd984128b2c294dc9c3b1a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:13.193079 2015 tablet_bootstrap.cc:492] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: Bootstrap starting.
I20260924 04:41:13.193243 2014 tablet_bootstrap.cc:492] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: Bootstrap starting.
I20260924 04:41:13.193840 2016 tablet_bootstrap.cc:492] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: Bootstrap starting.
I20260924 04:41:13.194132 2015 tablet_bootstrap.cc:654] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:13.194131 2014 tablet_bootstrap.cc:654] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:13.194738 2016 tablet_bootstrap.cc:654] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:13.196059 2015 tablet_bootstrap.cc:492] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: No bootstrap required, opened a new log
I20260924 04:41:13.196132 2015 ts_tablet_manager.cc:1397] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260924 04:41:13.196544 2015 raft_consensus.cc:348] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.196641 2015 raft_consensus.cc:374] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:13.196668 2015 raft_consensus.cc:729] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84759bf4644443c3a78d7f65b8804e19, State: Initialized, Role: FOLLOWER
I20260924 04:41:13.196774 2015 consensus_queue.cc:260] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.196992 2015 ts_tablet_manager.cc:1428] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:13.197050 2016 tablet_bootstrap.cc:492] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: No bootstrap required, opened a new log
I20260924 04:41:13.197129 2016 ts_tablet_manager.cc:1397] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:13.197217 2021 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.197319 2021 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.197599 2016 raft_consensus.cc:348] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.197734 2016 raft_consensus.cc:374] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:13.197767 2016 raft_consensus.cc:729] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f73a1a7a91484da32614d363918564, State: Initialized, Role: FOLLOWER
I20260924 04:41:13.197896 2016 consensus_queue.cc:260] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.198203 2016 ts_tablet_manager.cc:1428] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:13.198712 2022 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.198864 2022 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.204950 2014 tablet_bootstrap.cc:492] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: No bootstrap required, opened a new log
I20260924 04:41:13.205085 2014 ts_tablet_manager.cc:1397] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260924 04:41:13.205551 2014 raft_consensus.cc:348] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.205690 2014 raft_consensus.cc:374] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:13.205729 2014 raft_consensus.cc:729] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afd6625b4d94968957685aeb518647b, State: Initialized, Role: FOLLOWER
I20260924 04:41:13.205880 2014 consensus_queue.cc:260] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.206174 2014 ts_tablet_manager.cc:1428] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:41:13.206130 2020 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.206260 2020 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.236372 2020 raft_consensus.cc:482] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:13.236533 2020 raft_consensus.cc:504] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.237766 2020 leader_election.cc:288] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1:44851), 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203)
I20260924 04:41:13.242503 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.238175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34291 (local address 127.0.98.1:44851)
0924 04:41:13.238316 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.238320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.238620 (+ 300us) server_negotiation.cc:407] Connection header received
0924 04:41:13.238765 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.238767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.238818 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.238917 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:13.239451 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.239953 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.241373 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.241884 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.241963 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.242016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.242252 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.242370 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:41:13.242544 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.238047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44851 (local address 127.0.98.2:34291)
0924 04:41:13.238498 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:13.238514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:13.238647 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:13.238941 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:13.238944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:13.238987 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:13.239299 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.239308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.240076 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.240080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.241249 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.241259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.242001 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.242005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.242118 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.242144 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:41:13.242229 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:41:13.242355 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":90,"spinlock_wait_cycles":6784,"thread_start_us":162,"threads_started":1}
I20260924 04:41:13.242786 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.238076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44203 (local address 127.0.98.2:32997)
0924 04:41:13.238594 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:13.238606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:13.238689 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:13.238941 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:13.238944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:13.238986 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:13.239346 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.239357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.240105 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.240109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.241575 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.241585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.242336 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.242339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.242437 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.242459 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:41:13.242541 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:41:13.242611 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":156,"spinlock_wait_cycles":30336,"thread_start_us":121,"threads_started":1}
I20260924 04:41:13.242780 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.238207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32997 (local address 127.0.98.3:44203)
0924 04:41:13.238393 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.238398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.238686 (+ 288us) server_negotiation.cc:407] Connection header received
0924 04:41:13.238764 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.238767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.238835 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.238917 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:13.239464 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.239961 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.241704 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.242224 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.242317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.242373 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.242574 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.242654 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:41:13.243214 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ab4251fd984128b2c294dc9c3b1a18" candidate_uuid: "5afd6625b4d94968957685aeb518647b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84759bf4644443c3a78d7f65b8804e19" is_pre_election: true
I20260924 04:41:13.243343 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ab4251fd984128b2c294dc9c3b1a18" candidate_uuid: "5afd6625b4d94968957685aeb518647b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f73a1a7a91484da32614d363918564" is_pre_election: true
I20260924 04:41:13.243466 1945 raft_consensus.cc:2454] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5afd6625b4d94968957685aeb518647b in term 0.
I20260924 04:41:13.243475 1665 raft_consensus.cc:2454] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5afd6625b4d94968957685aeb518647b in term 0.
I20260924 04:41:13.243829 1738 leader_election.cc:302] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f73a1a7a91484da32614d363918564, 5afd6625b4d94968957685aeb518647b; no voters:
I20260924 04:41:13.244118 2020 raft_consensus.cc:2781] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:13.244175 2020 raft_consensus.cc:482] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:13.244203 2020 raft_consensus.cc:3037] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:13.245211 2020 raft_consensus.cc:504] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.245587 2020 leader_election.cc:288] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [CANDIDATE]: Term 1 election: Requested vote from peers 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1:44851), 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203)
I20260924 04:41:13.245963 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ab4251fd984128b2c294dc9c3b1a18" candidate_uuid: "5afd6625b4d94968957685aeb518647b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f73a1a7a91484da32614d363918564"
I20260924 04:41:13.245963 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ab4251fd984128b2c294dc9c3b1a18" candidate_uuid: "5afd6625b4d94968957685aeb518647b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84759bf4644443c3a78d7f65b8804e19"
I20260924 04:41:13.246084 1945 raft_consensus.cc:3037] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:13.246084 1665 raft_consensus.cc:3037] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:13.247279 1665 raft_consensus.cc:2454] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5afd6625b4d94968957685aeb518647b in term 1.
I20260924 04:41:13.247269 1945 raft_consensus.cc:2454] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5afd6625b4d94968957685aeb518647b in term 1.
I20260924 04:41:13.247584 1737 leader_election.cc:302] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5afd6625b4d94968957685aeb518647b, 84759bf4644443c3a78d7f65b8804e19; no voters:
I20260924 04:41:13.247767 2020 raft_consensus.cc:2781] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:13.248028 2020 raft_consensus.cc:686] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [term 1 LEADER]: Becoming Leader. State: Replica: 5afd6625b4d94968957685aeb518647b, State: Running, Role: LEADER
I20260924 04:41:13.248277 2020 consensus_queue.cc:237] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } }
I20260924 04:41:13.249482 2055 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: TxnStatusTablet state changed. Reason: New leader 5afd6625b4d94968957685aeb518647b. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.249604 2055 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:13.249840 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:41:13.249989 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:41:13.250835 1507 catalog_manager.cc:5495] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5afd6625b4d94968957685aeb518647b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:13.274277 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.270017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.0.1:51186)
0924 04:41:13.270238 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:13.270256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:13.270348 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:13.270685 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:13.270687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:13.270695 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:13.270783 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.270786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.271843 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.271845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.272439 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.272447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.273057 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.273060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.273125 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.273167 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.273953 (+ 786us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.273973 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:41:13.274058 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:41:13.274143 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 04:41:13.274277 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.270146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.0.98.2:46389)
0924 04:41:13.270298 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.270302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.270319 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:41:13.270498 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.270501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.270564 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.270672 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:13.270900 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.271714 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.272589 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.272956 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.273062 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.273305 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.273832 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.273945 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.274080 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.274135 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:41:13.277196 1945 raft_consensus.cc:1261] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Refusing update from remote peer 5afd6625b4d94968957685aeb518647b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:13.277408 1665 raft_consensus.cc:1261] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Refusing update from remote peer 5afd6625b4d94968957685aeb518647b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:13.277741 2020 consensus_queue.cc:1035] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:13.277944 2055 consensus_queue.cc:1035] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:13.279495 2022 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: TxnStatusTablet state changed. Reason: New leader 5afd6625b4d94968957685aeb518647b. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.279605 2022 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.279675 2021 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: TxnStatusTablet state changed. Reason: New leader 5afd6625b4d94968957685aeb518647b. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.279803 2021 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.281484 2020 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.281596 2020 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:13.281718 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332761080938270720 to 7332761080827498496. Current Snapshot: MvccSnapshot[applied={T|T < 7332761080938270720}]
I20260924 04:41:13.282119 2022 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.282229 2022 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 41f73a1a7a91484da32614d363918564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.284021 2055 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.284128 2055 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 5afd6625b4d94968957685aeb518647b: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:13.284778 2021 tablet_replica.cc:439] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5afd6625b4d94968957685aeb518647b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84759bf4644443c3a78d7f65b8804e19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44851 } } peers { permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 } } }
I20260924 04:41:13.284888 2021 tablet_replica.cc:442] T 43ab4251fd984128b2c294dc9c3b1a18 P 84759bf4644443c3a78d7f65b8804e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:13.285053 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332761080938270720 to 7332761080827498496. Current Snapshot: MvccSnapshot[applied={T|T < 7332761080938270720 or (T in {7332761080938270720})}]
I20260924 04:41:13.298182 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.294297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.1:56533)
0924 04:41:13.294695 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:13.294706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:13.294836 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:13.295140 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:13.295143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:13.295152 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:13.295259 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.295267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.296004 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.296011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.296653 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.296661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.297363 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.297366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.297461 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.297513 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.297851 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.297860 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:41:13.297937 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:41:13.298030 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":109,"threads_started":1}
I20260924 04:41:13.298171 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.294409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56533 (local address 127.0.98.2:46389)
0924 04:41:13.294556 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.294563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.294783 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:41:13.294949 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.294952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.295017 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.295097 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:13.295382 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.295891 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.296784 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.297133 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.297363 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.297663 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.297752 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.297836 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.297954 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.298015 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:41:13.305398 1945 raft_consensus.cc:1261] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Refusing update from remote peer 84759bf4644443c3a78d7f65b8804e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:13.305459 1805 raft_consensus.cc:1261] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Refusing update from remote peer 84759bf4644443c3a78d7f65b8804e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:13.305790 2025 consensus_queue.cc:1035] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f73a1a7a91484da32614d363918564" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:13.305944 2021 consensus_queue.cc:1035] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afd6625b4d94968957685aeb518647b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:13.329687 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.325739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46275 (local address 127.0.98.1:44851)
0924 04:41:13.325872 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.325877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.326061 (+ 184us) server_negotiation.cc:407] Connection header received
0924 04:41:13.326146 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.326149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.326201 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.326289 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:13.326505 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.327374 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.328432 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.328806 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.328925 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.329250 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.329333 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.329404 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.329504 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.329558 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:41:13.329770 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.325644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44851 (local address 127.0.98.2:46275)
0924 04:41:13.325974 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:13.325988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:13.326075 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:13.326289 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:13.326291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:13.326306 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:13.326398 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.326402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.327474 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.327477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.328318 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:13.328327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.328992 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.328995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:13.329070 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.329114 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:13.329418 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:13.329423 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:41:13.329476 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:41:13.329543 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":135,"threads_started":1}
I20260924 04:41:13.354869 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:13.355007 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:13.357048 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f94efb51e1a845bd9d2a19a80be49008: Sending DeleteTablet for 3 replicas of tablet 0dd8ce135615454cb51c59d6dedb1cd2
I20260924 04:41:13.357553 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd8ce135615454cb51c59d6dedb1cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60852
I20260924 04:41:13.357618 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd8ce135615454cb51c59d6dedb1cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51176
I20260924 04:41:13.357743 2038 tablet_replica.cc:330] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: stopping tablet replica
I20260924 04:41:13.357811 2039 tablet_replica.cc:330] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: stopping tablet replica
I20260924 04:41:13.357883 2038 raft_consensus.cc:2229] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:13.357913 2039 raft_consensus.cc:2229] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:13.358017 2039 raft_consensus.cc:2258] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:13.358072 2038 raft_consensus.cc:2258] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:13.358275 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd8ce135615454cb51c59d6dedb1cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44046
I20260924 04:41:13.358551 2037 tablet_replica.cc:330] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: stopping tablet replica
I20260924 04:41:13.358670 2037 raft_consensus.cc:2229] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:13.358808 2037 raft_consensus.cc:2258] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:13.359067 2039 ts_tablet_manager.cc:1905] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:13.359557 2037 ts_tablet_manager.cc:1905] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:13.359557 2038 ts_tablet_manager.cc:1905] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:13.361883 2039 ts_tablet_manager.cc:1918] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:13.361948 2039 log.cc:1196] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/0dd8ce135615454cb51c59d6dedb1cd2
I20260924 04:41:13.362193 2039 ts_tablet_manager.cc:1939] T 0dd8ce135615454cb51c59d6dedb1cd2 P 5afd6625b4d94968957685aeb518647b: Deleting consensus metadata
I20260924 04:41:13.362206 2037 ts_tablet_manager.cc:1918] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:13.362207 2038 ts_tablet_manager.cc:1918] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:13.362277 2037 log.cc:1196] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/0dd8ce135615454cb51c59d6dedb1cd2
I20260924 04:41:13.362277 2038 log.cc:1196] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/0dd8ce135615454cb51c59d6dedb1cd2
I20260924 04:41:13.362528 2038 ts_tablet_manager.cc:1939] T 0dd8ce135615454cb51c59d6dedb1cd2 P 84759bf4644443c3a78d7f65b8804e19: Deleting consensus metadata
I20260924 04:41:13.362528 2037 ts_tablet_manager.cc:1939] T 0dd8ce135615454cb51c59d6dedb1cd2 P 41f73a1a7a91484da32614d363918564: Deleting consensus metadata
I20260924 04:41:13.363322 1494 catalog_manager.cc:4841] TS 41f73a1a7a91484da32614d363918564 (127.0.98.3:44203): tablet 0dd8ce135615454cb51c59d6dedb1cd2 (table test-table [id=eecc77bc6a4046eebd763257bfe8eb11]) successfully deleted
I20260924 04:41:13.363317 1493 catalog_manager.cc:4841] TS 84759bf4644443c3a78d7f65b8804e19 (127.0.98.1:44851): tablet 0dd8ce135615454cb51c59d6dedb1cd2 (table test-table [id=eecc77bc6a4046eebd763257bfe8eb11]) successfully deleted
I20260924 04:41:13.363317 1492 catalog_manager.cc:4841] TS 5afd6625b4d94968957685aeb518647b (127.0.98.2:46389): tablet 0dd8ce135615454cb51c59d6dedb1cd2 (table test-table [id=eecc77bc6a4046eebd763257bfe8eb11]) successfully deleted
Sep 24 04:41:13 dist-test-slave-npg2 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:13.374725 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.363817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.0.98.1:44851)
0924 04:41:13.363966 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.363970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.364174 (+ 204us) server_negotiation.cc:407] Connection header received
0924 04:41:13.364310 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.364340 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.364385 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.364455 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:13.365273 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.365780 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.366890 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.367107 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.367255 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.370747 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:13.370757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:13.370769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:13.370795 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:13.372619 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:13.373175 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:13.373180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:13.373181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:13.373259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:13.373551 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:13.373553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:13.373554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:13.373891 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:13.374106 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.374413 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.374530 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 04:41:13.375222 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870
I20260924 04:41:13.385905 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:13.377055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.0.98.62:41575)
0924 04:41:13.377175 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:13.377181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:13.377247 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:41:13.377458 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:13.377462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:13.377524 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:13.377617 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:13.378446 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.379399 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.380235 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:13.380585 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:13.380752 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:13.382095 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:13.382100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:13.382104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:13.382135 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:13.384169 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:13.384579 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:13.384583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:13.384585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:13.384674 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:13.384952 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:13.384956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:13.384959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:13.385220 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:13.385363 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:13.385586 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:41:13.385719 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:41:13.386319 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45104
I20260924 04:41:13.387619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1580
I20260924 04:41:13.396423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1720
I20260924 04:41:13.403451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1860
I20260924 04:41:13.409106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 1476
2026-09-24T04:41:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2893 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:13 dist-test-slave-npg2 krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:13 dist-test-slave-npg2 krb5kdc[2078](info): set up 2 sockets
Sep 24 04:41:13 dist-test-slave-npg2 krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260924 04:41:14.463248 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:14 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:14.592257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:14.709059 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:14.709355 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:14.709396 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:14.713625 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:14.713709 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:41:14.713728 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:14.713747 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:14.713765 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:14.713780 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:14.719355 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:14.720516 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:14.721616 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:14.730728 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:14.730780 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:14.730780 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:14.731026 2091 server_base.cc:1034] running on GCE node
I20260924 04:41:14.731609 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:14.732756 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:14.733955 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790224874733920 us; error 50 us; skew 500 ppm
Sep 24 04:41:14 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:14.736896 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:14.737881 2091 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260924 04:41:14.738525 2091 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:14.738620 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:14.738848 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:14.741009 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "d7188e74b58847eeb5a329c95ee1f8c0"
format_stamp: "Formatted at 2026-09-24 04:41:14 on dist-test-slave-npg2"
server_key: "96cabf70b8515c0ba5694df03e368f56"
server_key_iv: "db1ac86418d4e86c2ed740b527846376"
server_key_version: "encryptionkey@0"
I20260924 04:41:14.741590 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7188e74b58847eeb5a329c95ee1f8c0"
format_stamp: "Formatted at 2026-09-24 04:41:14 on dist-test-slave-npg2"
server_key: "96cabf70b8515c0ba5694df03e368f56"
server_key_iv: "db1ac86418d4e86c2ed740b527846376"
server_key_version: "encryptionkey@0"
I20260924 04:41:14.745573 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:14.748121 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:14.749504 2091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:14.749607 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "d7188e74b58847eeb5a329c95ee1f8c0"
format_stamp: "Formatted at 2026-09-24 04:41:14 on dist-test-slave-npg2"
server_key: "96cabf70b8515c0ba5694df03e368f56"
server_key_iv: "db1ac86418d4e86c2ed740b527846376"
server_key_version: "encryptionkey@0"
I20260924 04:41:14.749722 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:14.780735 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:14.784372 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:14.784590 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:14.792886 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45549
I20260924 04:41:14.792910 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45549 every 8 connection(s)
I20260924 04:41:14.793871 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:14.796828 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:14.799248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 2091
I20260924 04:41:14.799350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:14.799631 392 external_mini_cluster.cc:1428] Setting key bce0955a927b76218f4367da141ca57c
I20260924 04:41:14.802984 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Bootstrap starting.
I20260924 04:41:14.805692 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:14.806540 2169 log.cc:824] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:14.808646 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: No bootstrap required, opened a new log
Sep 24 04:41:14 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:14.812554 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 04:41:14.812846 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:14.812913 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7188e74b58847eeb5a329c95ee1f8c0, State: Initialized, Role: FOLLOWER
I20260924 04:41:14.813395 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 04:41:14.813562 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:14.813632 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:14.813759 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:14.814697 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 04:41:14.814890 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:14.801164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.0.98.62:45549)
0924 04:41:14.801595 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:14.801611 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:14.801647 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:14.802268 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:14.802285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:14.802644 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:14.802949 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:14.803906 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:14.804963 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:14.806137 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:14.806645 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:14.806829 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:14.809766 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:14.809772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:14.809786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:14.809812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:14.811711 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:14.812201 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:14.812205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:14.812210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:14.812366 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:14.812661 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:14.812664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:14.812665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:14.813183 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:14.813389 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:14.813697 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:41:14.814043 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":157,"threads_started":1}
I20260924 04:41:14.815169 2169 leader_election.cc:302] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7188e74b58847eeb5a329c95ee1f8c0; no voters:
I20260924 04:41:14.815580 2169 leader_election.cc:288] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:14.815889 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:14.816221 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [term 1 LEADER]: Becoming Leader. State: Replica: d7188e74b58847eeb5a329c95ee1f8c0, State: Running, Role: LEADER
I20260924 04:41:14.816606 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260924 04:41:14.816924 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:14.818377 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 04:41:14.818437 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7188e74b58847eeb5a329c95ee1f8c0. Latest consensus state: current_term: 1 leader_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7188e74b58847eeb5a329c95ee1f8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260924 04:41:14.818511 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:14.818540 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:14.818899 2184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:14.822125 2184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:14.827968 2184 catalog_manager.cc:1347] Generated new cluster ID: 8ed838cd314a41f9a5066fa81890e39c
I20260924 04:41:14.828076 2184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:14.861474 2184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:14.862517 2184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:14.868523 2184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Generated new TSK 0
I20260924 04:41:14.869365 2184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:14.910991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:15.024976 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:15.025264 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:15.025312 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:15.029492 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:15.029560 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:15.029601 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:41:15.029615 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:15.029703 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:15.035301 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:15.036494 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:15.037601 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:15.047596 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:15.047643 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:15.047885 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:15.048054 2195 server_base.cc:1034] running on GCE node
I20260924 04:41:15.048496 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:15.049245 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:15.050442 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790224875050427 us; error 30 us; skew 500 ppm
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:15.053541 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:15.054558 2195 webserver.cc:460] Webserver started at http://127.0.98.1:42483/ using document root <none> and password file <none>
I20260924 04:41:15.055086 2195 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:15.055219 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:15.055433 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:15.057500 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "d868ba0d1f394806804df63e78f870ad"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "3ddbc3f2e56f52e4c1bc7dbeb217b562"
server_key_iv: "4f1d9e24c999c81c88de39f25b9dcd2f"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.058033 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d868ba0d1f394806804df63e78f870ad"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "3ddbc3f2e56f52e4c1bc7dbeb217b562"
server_key_iv: "4f1d9e24c999c81c88de39f25b9dcd2f"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.061843 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:41:15.064440 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:15.065737 2195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:15.065878 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "d868ba0d1f394806804df63e78f870ad"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "3ddbc3f2e56f52e4c1bc7dbeb217b562"
server_key_iv: "4f1d9e24c999c81c88de39f25b9dcd2f"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.065999 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:15.095095 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:15.098860 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:15.099081 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:15.100656 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:15.100736 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:15.100808 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:15.100859 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:15.111810 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37813
I20260924 04:41:15.111896 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37813 every 8 connection(s)
I20260924 04:41:15.112933 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:15.114972 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.101516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60825 (local address 127.0.98.62:45549)
0924 04:41:15.101645 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.101648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.102751 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 04:41:15.103859 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.103864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.103928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.104028 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.105390 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.106043 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.107388 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.107626 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.107747 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.111220 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.111224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.111231 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.111288 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.112985 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.113577 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.113580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.113583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.113667 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.113997 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.114000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.114001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.114177 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.114255 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.114710 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.114850 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:15.115895 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.102092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:60825)
0924 04:41:15.102594 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.102644 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.103592 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.104284 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.104295 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.104674 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.105187 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.105236 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.106175 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.106179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.107262 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.107275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.107800 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.107804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.107981 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.108632 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.108648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.110994 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.113142 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.113147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.113159 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.113462 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.113759 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.113761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.113763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.113885 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.114280 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.114285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.114502 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.114668 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.114948 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260924 04:41:15.117238 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 04:41:15.117450 2326 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:15.118011 2326 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 04:41:15.118427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 2195
I20260924 04:41:15.118559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:15.118824 392 external_mini_cluster.cc:1428] Setting key 17f1e9d8cf4578ceeb965794983d9f48
I20260924 04:41:15.120669 2123 ts_manager.cc:194] Registered new tserver with Master: d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813)
I20260924 04:41:15.122246 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60825
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:15.130594 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.119755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:47285)
0924 04:41:15.122932 (+ 3177us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.122947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.123052 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.123465 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.123468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.123675 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.123814 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.123821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.124597 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.124600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.125212 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.125221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.125680 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.125684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.125790 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.126397 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.126410 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.126771 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.128869 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.128875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.128877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.129097 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.129469 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.129474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.129476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.129575 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.129983 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.129986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.130092 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.130214 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.130383 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3056,"thread_start_us":96,"threads_started":1}
I20260924 04:41:15.130609 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.120183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47285 (local address 127.0.98.62:45549)
0924 04:41:15.120286 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.120291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.123032 (+ 2741us) server_negotiation.cc:407] Connection header received
0924 04:41:15.123171 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.123178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.123315 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.123445 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.123944 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.124464 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.125337 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.125561 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.125673 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.126947 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.126951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.126955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.126983 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.128752 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.129208 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.129214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.129216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.129302 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.129684 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.129688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.129690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.129881 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.129995 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.130270 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.130421 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:15.147197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:15.260659 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:15.261040 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:15.261139 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:15.265246 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:15.265355 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:15.265429 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:41:15.265475 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:15.265537 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:15.271145 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:15.272265 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:15.273360 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:15.283617 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:15.283622 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:15.283787 2335 server_base.cc:1034] running on GCE node
W20260924 04:41:15.283619 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:15.284420 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:15.285384 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:15.286607 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790224875286572 us; error 72 us; skew 500 ppm
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:15.290198 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:15.291307 2335 webserver.cc:460] Webserver started at http://127.0.98.2:44881/ using document root <none> and password file <none>
I20260924 04:41:15.292114 2335 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:15.292268 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:15.292593 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:15.294883 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c00f1753df84472a39817cbce7b7b5c"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "d8030e2c888f1cec6b32bbb68a2acb80"
server_key_iv: "dee6f450af34aaf0bcab1861f1132ce5"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.295557 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c00f1753df84472a39817cbce7b7b5c"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "d8030e2c888f1cec6b32bbb68a2acb80"
server_key_iv: "dee6f450af34aaf0bcab1861f1132ce5"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.299773 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:15.302435 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:15.303974 2335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:15.304078 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "7c00f1753df84472a39817cbce7b7b5c"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "d8030e2c888f1cec6b32bbb68a2acb80"
server_key_iv: "dee6f450af34aaf0bcab1861f1132ce5"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.304173 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:15.318640 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:15.321947 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:15.322147 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:15.323757 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:15.323807 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:15.323879 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:15.323910 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:15.334806 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46225
I20260924 04:41:15.334839 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46225 every 8 connection(s)
I20260924 04:41:15.335783 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:15.338644 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.324702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48697 (local address 127.0.98.62:45549)
0924 04:41:15.324840 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.324844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.325972 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 04:41:15.327157 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.327160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.327217 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.327306 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.328942 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.329599 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.330754 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.330991 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.331113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.334786 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.334791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.334794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.334828 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.336707 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.337241 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.337244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.337245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.337335 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.337656 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.337658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.337659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.337838 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.337925 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.338398 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.338502 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:41:15.339602 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.325286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:48697)
0924 04:41:15.325805 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.325849 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.326902 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.327545 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.327557 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.328017 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.328735 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.328754 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.329727 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.329731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.330622 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.330631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.331204 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.331209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.331386 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.332681 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.332697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.334526 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.336803 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.336809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.336823 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.337128 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.337436 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.337438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.337440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.337573 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.337941 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.337946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.338189 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.338355 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.338605 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":134,"threads_started":1}
I20260924 04:41:15.341069 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 04:41:15.341220 2466 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:15.341851 2466 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 04:41:15.343055 2122 ts_manager.cc:194] Registered new tserver with Master: 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225)
I20260924 04:41:15.343997 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48697
I20260924 04:41:15.344040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 2335
I20260924 04:41:15.344156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:15.344471 392 external_mini_cluster.cc:1428] Setting key f2292406a2a536c64118919ca000e1aa
I20260924 04:41:15.352938 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.343410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:39849)
0924 04:41:15.343833 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.343848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.344007 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.344401 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.344403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.344662 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.344800 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.344808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.346058 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.346062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.346714 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.346723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.347466 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.347469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.347543 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.348174 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.348195 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.348714 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.350919 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.350923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.350924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.351186 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.351550 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.351552 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.351553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.351632 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.352282 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.352286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.352391 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.352521 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.352765 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":131,"threads_started":1}
I20260924 04:41:15.353050 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.343540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39849 (local address 127.0.98.62:45549)
0924 04:41:15.343660 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.343665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.343932 (+ 267us) server_negotiation.cc:407] Connection header received
0924 04:41:15.344144 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.344147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.344191 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.344246 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.344930 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.345825 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.346851 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.347326 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.347464 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.348882 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.348885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.348888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.348912 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.350774 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.351289 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.351296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.351300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.351437 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.351732 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.351739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.351745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.352129 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.352266 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.352632 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.352794 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:15.374926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:15.492786 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:15.493086 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:15.493165 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:15.497195 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:15.497303 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:15.497373 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:41:15.497417 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:15.497488 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:15.502988 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:15.504130 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:15.505162 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:15.515058 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:15.515067 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:15.515072 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:15.515480 2475 server_base.cc:1034] running on GCE node
I20260924 04:41:15.515951 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:15.516667 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:15.517903 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790224875517881 us; error 32 us; skew 500 ppm
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:15.521169 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:15.522082 2475 webserver.cc:460] Webserver started at http://127.0.98.3:45125/ using document root <none> and password file <none>
I20260924 04:41:15.522673 2475 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:15.522814 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:15.523051 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:15.525177 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "1595f623b10a40c982c16398b66507ee"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "7a7db4941af4585578bf77dd14c648f8"
server_key_iv: "9991c01aa9117727ccbfb633846d17de"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.525754 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1595f623b10a40c982c16398b66507ee"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "7a7db4941af4585578bf77dd14c648f8"
server_key_iv: "9991c01aa9117727ccbfb633846d17de"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.529686 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:15.532433 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:15.533650 2475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:15.533792 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "1595f623b10a40c982c16398b66507ee"
format_stamp: "Formatted at 2026-09-24 04:41:15 on dist-test-slave-npg2"
server_key: "7a7db4941af4585578bf77dd14c648f8"
server_key_iv: "9991c01aa9117727ccbfb633846d17de"
server_key_version: "encryptionkey@0"
I20260924 04:41:15.533921 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:15.554594 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:15.557808 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:15.558027 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:15.559686 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:15.559760 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:15.559826 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:15.559861 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:15.570746 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43975
I20260924 04:41:15.570778 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43975 every 8 connection(s)
I20260924 04:41:15.571903 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:15.573726 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.560308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45659 (local address 127.0.98.62:45549)
0924 04:41:15.560465 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.560471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.561368 (+ 897us) server_negotiation.cc:407] Connection header received
0924 04:41:15.562420 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.562424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.562478 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.562537 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.564176 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.564990 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.566363 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.566794 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.566936 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.569796 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.569799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.569802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.569830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.571616 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.572135 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.572139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.572141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.572240 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.572663 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.572667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.572669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.572937 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.573031 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.573514 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.573611 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:15.574589 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.560742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:45659)
0924 04:41:15.561204 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.561254 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.562180 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.563005 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.563014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.563453 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.564020 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.564039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.565120 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.565123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.566224 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.566233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.566986 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.566990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.567241 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.567885 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.567901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.569600 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.571739 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.571744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.571755 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.572032 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.572349 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.572352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.572354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.572533 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.573046 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.573052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.573306 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.573470 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.573711 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":133,"threads_started":1}
I20260924 04:41:15.575884 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260924 04:41:15.576056 2606 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:15.576551 2606 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260924 04:41:15.577665 2123 ts_manager.cc:194] Registered new tserver with Master: 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:15.578496 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45659
I20260924 04:41:15.581684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 2475
I20260924 04:41:15.581805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:15.582077 392 external_mini_cluster.cc:1428] Setting key 50579ebe30de727f52955df73eec62d2
I20260924 04:41:15.584357 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:41:15.586117 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.578123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46303 (local address 127.0.98.62:45549)
0924 04:41:15.578253 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.578257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.578480 (+ 223us) server_negotiation.cc:407] Connection header received
0924 04:41:15.578612 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.578615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.578667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.578754 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.579286 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.580052 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.580946 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.581155 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.581238 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.582684 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.582690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.582694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.582720 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.584426 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.584954 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.584957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.584958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.585031 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.585333 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.585335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.585336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.585502 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.585576 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.585911 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.586004 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:41:15.586302 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.578058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:46303)
0924 04:41:15.578393 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.578407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.578503 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.578775 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.578778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.579003 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.579167 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.579176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.580243 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.580247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.580816 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.580825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.581267 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.581270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.581498 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.582127 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.582141 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.582562 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.584561 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.584565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.584567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.584824 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.585124 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.585127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.585130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.585238 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.585598 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.585603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.585721 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.585846 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.586019 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:15.611157 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.602579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.0.98.62:45549)
0924 04:41:15.602715 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.602719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.602732 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:41:15.602771 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.602773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.602810 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.602936 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.603709 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.604234 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.605132 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.605346 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.605453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.608003 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.608007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.608009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.608036 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.609466 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.609879 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.609883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.609885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.609968 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.610239 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.610244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.610246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.610433 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.610578 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.610808 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.610958 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:15.614311 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:15.617528 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:15.633227 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.626708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.0.1:34350)
0924 04:41:15.628389 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.628406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.628500 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.628960 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.628963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.628983 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:15.629079 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.629085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.630389 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.630392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.631485 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.631494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.632565 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.632573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.632702 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.632743 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.632883 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.633041 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260924 04:41:15.633426 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.626531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.98.1:37813)
0924 04:41:15.627583 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.627591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.627615 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:41:15.627699 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.627708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.627954 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.628149 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:15.628526 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.629572 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.631681 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.632577 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.632668 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.632781 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.633188 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.633280 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":95,"threads_started":1}
I20260924 04:41:15.633771 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.627181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.98.2:46225)
0924 04:41:15.627934 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.627943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.628475 (+ 532us) server_negotiation.cc:407] Connection header received
0924 04:41:15.628613 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.628621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.628789 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.628941 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:15.629203 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.630268 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.631601 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.632427 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.632547 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.632725 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.632922 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.633030 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":91,"threads_started":1}
I20260924 04:41:15.633811 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.626229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.0.1:33438)
0924 04:41:15.627047 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.627100 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.627284 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.628134 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.628141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.628176 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:15.628344 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.628360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.629690 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.629697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.631520 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.631526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.632870 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.632876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.633030 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.633053 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.633612 (+ 559us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.633682 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":181,"thread_start_us":101,"threads_started":1}
I20260924 04:41:15.634327 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.626489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43975 (local address 127.0.0.1:48940)
0924 04:41:15.627573 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.627588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.627686 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.628293 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.628297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.628314 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:15.628427 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.628433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.629841 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.629845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.631512 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.631524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.632885 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.632890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.633069 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.633093 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.633167 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.633228 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":129,"thread_start_us":125,"threads_started":1}
I20260924 04:41:15.634809 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.627110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.0.98.3:43975)
0924 04:41:15.627568 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.627581 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.627655 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:41:15.628019 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.628023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.628156 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.628311 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:15.628551 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.629727 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.631668 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.632517 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.632632 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.632816 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.633306 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.634493 (+ 1187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":108,"threads_started":1}
I20260924 04:41:15.636821 2260 tablet_service.cc:1467] Processing CreateTablet for tablet bc542dfff9644a8e912f5cee8afac072 (DEFAULT_TABLE table=test-table [id=2edff08e81c545e59475f6f3fe375ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:15.636917 2400 tablet_service.cc:1467] Processing CreateTablet for tablet bc542dfff9644a8e912f5cee8afac072 (DEFAULT_TABLE table=test-table [id=2edff08e81c545e59475f6f3fe375ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:15.637461 2540 tablet_service.cc:1467] Processing CreateTablet for tablet bc542dfff9644a8e912f5cee8afac072 (DEFAULT_TABLE table=test-table [id=2edff08e81c545e59475f6f3fe375ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:15.637949 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc542dfff9644a8e912f5cee8afac072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:15.638002 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc542dfff9644a8e912f5cee8afac072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:15.638444 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc542dfff9644a8e912f5cee8afac072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:15.644320 2629 tablet_bootstrap.cc:492] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Bootstrap starting.
I20260924 04:41:15.646359 2631 tablet_bootstrap.cc:492] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Bootstrap starting.
I20260924 04:41:15.646595 2629 tablet_bootstrap.cc:654] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:15.647451 2630 tablet_bootstrap.cc:492] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Bootstrap starting.
I20260924 04:41:15.647603 2629 log.cc:824] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:15.648519 2631 tablet_bootstrap.cc:654] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:15.649292 2631 log.cc:824] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:15.649617 2630 tablet_bootstrap.cc:654] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:15.649667 2629 tablet_bootstrap.cc:492] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: No bootstrap required, opened a new log
I20260924 04:41:15.649870 2629 ts_tablet_manager.cc:1397] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:41:15.650434 2630 log.cc:824] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:15.651502 2631 tablet_bootstrap.cc:492] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: No bootstrap required, opened a new log
I20260924 04:41:15.651695 2631 ts_tablet_manager.cc:1397] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:41:15.652257 2630 tablet_bootstrap.cc:492] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: No bootstrap required, opened a new log
I20260924 04:41:15.652469 2630 ts_tablet_manager.cc:1397] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:41:15.654563 2629 raft_consensus.cc:348] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.654836 2629 raft_consensus.cc:374] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:15.654904 2629 raft_consensus.cc:729] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d868ba0d1f394806804df63e78f870ad, State: Initialized, Role: FOLLOWER
I20260924 04:41:15.655388 2629 consensus_queue.cc:260] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.655721 2631 raft_consensus.cc:348] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.655949 2631 raft_consensus.cc:374] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:15.656010 2631 raft_consensus.cc:729] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1595f623b10a40c982c16398b66507ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:15.656244 2326 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260924 04:41:15.656374 2629 ts_tablet_manager.cc:1428] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:41:15.656436 2631 consensus_queue.cc:260] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.656468 2630 raft_consensus.cc:348] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.656714 2630 raft_consensus.cc:374] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:15.656776 2630 raft_consensus.cc:729] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Initialized, Role: FOLLOWER
I20260924 04:41:15.657166 2606 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260924 04:41:15.657357 2631 ts_tablet_manager.cc:1428] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:41:15.658267 2630 consensus_queue.cc:260] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.658965 2466 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260924 04:41:15.659323 2630 ts_tablet_manager.cc:1428] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260924 04:41:15.823069 2607 tablet.cc:2369] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:15.837134 2467 tablet.cc:2369] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:15.864362 2327 tablet.cc:2369] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:15.877218 2637 raft_consensus.cc:482] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:15.877460 2637 raft_consensus.cc:504] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.878603 2637 leader_election.cc:288] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:15.883060 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.879041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38829 (local address 127.0.98.3:43975)
0924 04:41:15.879221 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.879226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.879333 (+ 107us) server_negotiation.cc:407] Connection header received
0924 04:41:15.879452 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.879455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.879520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.879601 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:15.880004 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.880589 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.881936 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.882427 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.882557 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.882621 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.882856 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.882927 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 04:41:15.883083 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.878902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43975 (local address 127.0.98.2:38829)
0924 04:41:15.879245 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.879258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.879348 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.879631 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.879634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.879672 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:15.879875 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.879883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.880731 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.880737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.881802 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.881814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.882574 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.882578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.882689 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.882740 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.882839 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.882927 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":43,"spinlock_wait_cycles":33024,"thread_start_us":93,"threads_started":1}
I20260924 04:41:15.883494 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.878994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34097 (local address 127.0.98.1:37813)
0924 04:41:15.879207 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.879212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.879333 (+ 121us) server_negotiation.cc:407] Connection header received
0924 04:41:15.879452 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.879455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.879512 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.879601 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:15.880761 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.881258 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.882439 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.882861 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.882982 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.883041 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.883296 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.883386 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 04:41:15.883869 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.878878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.98.2:34097)
0924 04:41:15.879199 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.879215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.879342 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.879631 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.879634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.879674 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:15.879882 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.879888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.881366 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.881370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.882305 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.882314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.882983 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.882986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.883083 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.883109 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.883680 (+ 571us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.883749 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":54,"spinlock_wait_cycles":7808,"thread_start_us":88,"threads_started":1}
I20260924 04:41:15.883790 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc542dfff9644a8e912f5cee8afac072" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee" is_pre_election: true
I20260924 04:41:15.884162 2560 raft_consensus.cc:2454] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 0.
I20260924 04:41:15.884513 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc542dfff9644a8e912f5cee8afac072" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad" is_pre_election: true
I20260924 04:41:15.884686 2351 leader_election.cc:302] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, 7c00f1753df84472a39817cbce7b7b5c; no voters:
I20260924 04:41:15.884771 2280 raft_consensus.cc:2454] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 0.
I20260924 04:41:15.885008 2637 raft_consensus.cc:2781] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:15.885113 2637 raft_consensus.cc:482] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:15.885171 2637 raft_consensus.cc:3037] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:15.886219 2637 raft_consensus.cc:504] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.886601 2637 leader_election.cc:288] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 election: Requested vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:15.886911 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc542dfff9644a8e912f5cee8afac072" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad"
I20260924 04:41:15.886924 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc542dfff9644a8e912f5cee8afac072" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee"
I20260924 04:41:15.887061 2280 raft_consensus.cc:3037] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:15.887054 2560 raft_consensus.cc:3037] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:15.888216 2280 raft_consensus.cc:2454] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 1.
I20260924 04:41:15.888509 2354 leader_election.cc:302] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c00f1753df84472a39817cbce7b7b5c, d868ba0d1f394806804df63e78f870ad; no voters:
I20260924 04:41:15.888669 2637 raft_consensus.cc:2781] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:15.888811 2560 raft_consensus.cc:2454] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 1.
I20260924 04:41:15.888846 2637 raft_consensus.cc:686] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 1 LEADER]: Becoming Leader. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Running, Role: LEADER
I20260924 04:41:15.889226 2637 consensus_queue.cc:237] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } }
I20260924 04:41:15.892448 2118 catalog_manager.cc:5495] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } health_report { overall_health: HEALTHY } } }
Sep 24 04:41:15 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:15.949927 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.940888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.0.1:40478)
0924 04:41:15.941197 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:15.941209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:15.941372 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:15.941635 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:15.941637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:15.942094 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:15.942192 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.942198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.943025 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.943028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.943749 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:15.943759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.944276 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.944285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:15.944369 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.944939 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:15.944950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:15.946604 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:15.948395 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.948403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.948409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.948659 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.948942 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:15.948944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:15.948946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:15.949030 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:15.949403 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:15.949410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:15.949527 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 04:41:15.949644 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:41:15.949776 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":134,"threads_started":1}
I20260924 04:41:15.950021 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:15.940970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.0.98.62:45549)
0924 04:41:15.941112 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:15.941117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:15.941339 (+ 222us) server_negotiation.cc:407] Connection header received
0924 04:41:15.941462 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:15.941465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:15.941504 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:15.941566 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:15.942417 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.942920 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.943869 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:15.944075 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:15.944156 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:15.946753 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:15.946756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:15.946758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:15.946778 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:15.948286 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.948762 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.948764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.948765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.948833 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:15.949115 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:15.949117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:15.949119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:15.949284 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:15.949381 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:15.949756 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 04:41:15.949868 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:41:15.954787 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40478:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:41:15.955587 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:15.960523 2260 tablet_service.cc:1467] Processing CreateTablet for tablet f07d03cae4324e9aa807a933c4a18b85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ba4b9fadad043e9bbd41b9378fec461]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:15.960515 2400 tablet_service.cc:1467] Processing CreateTablet for tablet f07d03cae4324e9aa807a933c4a18b85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ba4b9fadad043e9bbd41b9378fec461]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:15.960598 2540 tablet_service.cc:1467] Processing CreateTablet for tablet f07d03cae4324e9aa807a933c4a18b85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ba4b9fadad043e9bbd41b9378fec461]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:15.961251 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07d03cae4324e9aa807a933c4a18b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:15.961246 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07d03cae4324e9aa807a933c4a18b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:15.961246 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07d03cae4324e9aa807a933c4a18b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:15.963798 2629 tablet_bootstrap.cc:492] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: Bootstrap starting.
I20260924 04:41:15.964083 2631 tablet_bootstrap.cc:492] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: Bootstrap starting.
I20260924 04:41:15.964250 2630 tablet_bootstrap.cc:492] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: Bootstrap starting.
I20260924 04:41:15.964875 2629 tablet_bootstrap.cc:654] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:15.964943 2631 tablet_bootstrap.cc:654] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:15.965525 2630 tablet_bootstrap.cc:654] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:15.966275 2629 tablet_bootstrap.cc:492] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: No bootstrap required, opened a new log
I20260924 04:41:15.966370 2629 ts_tablet_manager.cc:1397] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:15.966425 2631 tablet_bootstrap.cc:492] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: No bootstrap required, opened a new log
I20260924 04:41:15.966528 2631 ts_tablet_manager.cc:1397] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:15.966826 2630 tablet_bootstrap.cc:492] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: No bootstrap required, opened a new log
I20260924 04:41:15.966919 2630 ts_tablet_manager.cc:1397] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:15.966879 2629 raft_consensus.cc:348] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:15.966943 2631 raft_consensus.cc:348] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:15.967008 2629 raft_consensus.cc:374] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:15.967058 2629 raft_consensus.cc:729] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d868ba0d1f394806804df63e78f870ad, State: Initialized, Role: FOLLOWER
I20260924 04:41:15.967064 2631 raft_consensus.cc:374] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:15.967100 2631 raft_consensus.cc:729] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1595f623b10a40c982c16398b66507ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:15.967216 2629 consensus_queue.cc:260] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:15.967258 2631 consensus_queue.cc:260] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:15.967487 2629 ts_tablet_manager.cc:1428] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:41:15.967428 2630 raft_consensus.cc:348] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:15.967535 2630 raft_consensus.cc:374] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:15.967567 2630 raft_consensus.cc:729] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Initialized, Role: FOLLOWER
I20260924 04:41:15.967563 2631 ts_tablet_manager.cc:1428] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:15.967690 2630 consensus_queue.cc:260] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:15.967703 2635 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:15.967820 2635 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:15.967803 2636 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:15.967948 2636 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:15.967950 2630 ts_tablet_manager.cc:1428] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:15.968127 2637 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:15.968237 2637 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:16.109555 2637 raft_consensus.cc:482] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:16.109694 2637 raft_consensus.cc:504] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.110176 2637 leader_election.cc:288] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:16.110705 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07d03cae4324e9aa807a933c4a18b85" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad" is_pre_election: true
I20260924 04:41:16.110705 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07d03cae4324e9aa807a933c4a18b85" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee" is_pre_election: true
I20260924 04:41:16.110875 2280 raft_consensus.cc:2454] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 0.
I20260924 04:41:16.110909 2560 raft_consensus.cc:2454] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 0.
I20260924 04:41:16.111259 2351 leader_election.cc:302] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, 7c00f1753df84472a39817cbce7b7b5c; no voters:
I20260924 04:41:16.111465 2637 raft_consensus.cc:2781] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:16.111522 2637 raft_consensus.cc:482] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:16.111575 2637 raft_consensus.cc:3037] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:16.112857 2637 raft_consensus.cc:504] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.113194 2637 leader_election.cc:288] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 election: Requested vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:16.113538 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07d03cae4324e9aa807a933c4a18b85" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee"
I20260924 04:41:16.113643 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07d03cae4324e9aa807a933c4a18b85" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad"
I20260924 04:41:16.113682 2560 raft_consensus.cc:3037] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:16.113771 2280 raft_consensus.cc:3037] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:16.114796 2560 raft_consensus.cc:2454] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 1.
I20260924 04:41:16.115192 2351 leader_election.cc:302] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, 7c00f1753df84472a39817cbce7b7b5c; no voters:
I20260924 04:41:16.115350 2637 raft_consensus.cc:2781] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:16.115439 2637 raft_consensus.cc:686] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [term 1 LEADER]: Becoming Leader. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Running, Role: LEADER
I20260924 04:41:16.115576 2280 raft_consensus.cc:2454] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 1.
I20260924 04:41:16.115607 2637 consensus_queue.cc:237] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.116377 2637 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: TxnStatusTablet state changed. Reason: New leader 7c00f1753df84472a39817cbce7b7b5c. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.116466 2637 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:16.117225 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:41:16.117443 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:41:16.117484 2123 catalog_manager.cc:5495] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:16.163985 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.159809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.0.98.2:46225)
0924 04:41:16.160055 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.160066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.160079 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:41:16.160146 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.160148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.160197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.160290 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:16.160558 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.161102 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.162398 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.162643 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.162725 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.163049 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.163544 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.163654 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.163796 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.163845 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":111,"threads_started":1}
I20260924 04:41:16.163980 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.159681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.0.1:34354)
0924 04:41:16.159857 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:16.159874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:16.159978 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:16.160332 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:16.160335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:16.160343 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:16.160451 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.160457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.161204 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.161207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.162259 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.162279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.162749 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.162753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.162843 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.162897 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.163637 (+ 740us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.163652 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 04:41:16.163745 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:41:16.163832 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:41:16.169234 2560 raft_consensus.cc:1261] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Refusing update from remote peer 7c00f1753df84472a39817cbce7b7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:16.169234 2280 raft_consensus.cc:1261] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Refusing update from remote peer 7c00f1753df84472a39817cbce7b7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:16.169953 2637 consensus_queue.cc:1035] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:16.170154 2640 consensus_queue.cc:1035] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:16.174098 2636 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: TxnStatusTablet state changed. Reason: New leader 7c00f1753df84472a39817cbce7b7b5c. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.174211 2636 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:16.174714 2635 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: TxnStatusTablet state changed. Reason: New leader 7c00f1753df84472a39817cbce7b7b5c. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.174835 2635 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:16.177459 2640 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.177595 2640 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:16.177745 2656 mvcc.cc:205] Tried to move back new op lower bound from 7332761092781940736 to 7332761092571168768. Current Snapshot: MvccSnapshot[applied={T|T < 7332761092781940736}]
I20260924 04:41:16.178138 2636 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.178264 2636 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 1595f623b10a40c982c16398b66507ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:16.178294 2657 mvcc.cc:205] Tried to move back new op lower bound from 7332761092781940736 to 7332761092571168768. Current Snapshot: MvccSnapshot[applied={T|T < 7332761092781940736}]
I20260924 04:41:16.178504 2635 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.178599 2635 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P d868ba0d1f394806804df63e78f870ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:16.180893 2637 tablet_replica.cc:439] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } } }
I20260924 04:41:16.181007 2637 tablet_replica.cc:442] T f07d03cae4324e9aa807a933c4a18b85 P 7c00f1753df84472a39817cbce7b7b5c: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:16.184054 2658 mvcc.cc:205] Tried to move back new op lower bound from 7332761092781940736 to 7332761092571168768. Current Snapshot: MvccSnapshot[applied={T|T < 7332761092781940736}]
I20260924 04:41:16.188069 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.184121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.0.98.2:46225)
0924 04:41:16.184249 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.184253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.184490 (+ 237us) server_negotiation.cc:407] Connection header received
0924 04:41:16.184687 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.184690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.184736 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.184781 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:16.185168 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.185901 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.186967 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.187286 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.187398 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.187657 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.187730 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.187809 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.187895 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.187945 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:41:16.198438 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.195255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56079 (local address 127.0.98.2:46225)
0924 04:41:16.195374 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.195378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.195532 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:41:16.195671 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.195675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.195719 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.195796 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:16.196034 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.196523 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.197359 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.197555 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.197676 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.197980 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.198045 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.198123 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.198211 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.198287 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260924 04:41:16.198448 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.195198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.98.2:56079)
0924 04:41:16.195454 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:16.195465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:16.195578 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:16.195801 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:16.195804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:16.195810 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:16.195937 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.195943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.196631 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.196634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.197230 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.197238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.197689 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.197692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.197783 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.197818 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.198123 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.198130 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:41:16.198203 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:41:16.198308 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260924 04:41:16.204881 2280 raft_consensus.cc:1261] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Refusing update from remote peer 7c00f1753df84472a39817cbce7b7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:16.204881 2560 raft_consensus.cc:1261] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Refusing update from remote peer 7c00f1753df84472a39817cbce7b7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:16.205235 2640 consensus_queue.cc:1035] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:16.205359 2637 consensus_queue.cc:1035] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:16.250129 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:16.250332 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:16.252815 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Sending DeleteTablet for 3 replicas of tablet bc542dfff9644a8e912f5cee8afac072
I20260924 04:41:16.253574 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet bc542dfff9644a8e912f5cee8afac072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33438
I20260924 04:41:16.253611 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet bc542dfff9644a8e912f5cee8afac072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48940
I20260924 04:41:16.254292 2675 tablet_replica.cc:330] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: stopping tablet replica
I20260924 04:41:16.254274 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:41:16.254666 2675 raft_consensus.cc:2229] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:41:16.254695 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:16.254918 2675 raft_consensus.cc:2258] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:16.255505 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet bc542dfff9644a8e912f5cee8afac072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34350
I20260924 04:41:16.256332 2676 tablet_replica.cc:330] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: stopping tablet replica
I20260924 04:41:16.256460 2675 ts_tablet_manager.cc:1905] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:16.256659 2676 raft_consensus.cc:2229] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:16.257045 2676 raft_consensus.cc:2258] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:16.258324 2676 ts_tablet_manager.cc:1905] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:16.259361 2674 tablet_replica.cc:330] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: stopping tablet replica
I20260924 04:41:16.259677 2674 raft_consensus.cc:2229] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:16.259948 2674 raft_consensus.cc:2258] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:16.260907 2675 ts_tablet_manager.cc:1918] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:16.261024 2675 log.cc:1196] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/bc542dfff9644a8e912f5cee8afac072
I20260924 04:41:16.261375 2675 ts_tablet_manager.cc:1939] T bc542dfff9644a8e912f5cee8afac072 P d868ba0d1f394806804df63e78f870ad: Deleting consensus metadata
I20260924 04:41:16.261593 2676 ts_tablet_manager.cc:1918] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:16.261680 2676 log.cc:1196] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/bc542dfff9644a8e912f5cee8afac072
I20260924 04:41:16.261988 2676 ts_tablet_manager.cc:1939] T bc542dfff9644a8e912f5cee8afac072 P 7c00f1753df84472a39817cbce7b7b5c: Deleting consensus metadata
I20260924 04:41:16.262298 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 34bbed415f9e4c5ebf7dc94e1db3469d (DEFAULT_TABLE table=test-table [id=1e070531d7584b8db4930c2e1492790c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:16.262574 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bbed415f9e4c5ebf7dc94e1db3469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:16.262696 2110 catalog_manager.cc:4841] TS d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813): tablet bc542dfff9644a8e912f5cee8afac072 (table test-table [id=2edff08e81c545e59475f6f3fe375ec3]) successfully deleted
I20260924 04:41:16.263636 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 34bbed415f9e4c5ebf7dc94e1db3469d (DEFAULT_TABLE table=test-table [id=1e070531d7584b8db4930c2e1492790c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:16.263895 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bbed415f9e4c5ebf7dc94e1db3469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:16.264837 2631 tablet_bootstrap.cc:492] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Bootstrap starting.
I20260924 04:41:16.265789 2631 tablet_bootstrap.cc:654] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:16.266224 2629 tablet_bootstrap.cc:492] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Bootstrap starting.
I20260924 04:41:16.266625 2109 catalog_manager.cc:4841] TS 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225): tablet bc542dfff9644a8e912f5cee8afac072 (table test-table [id=2edff08e81c545e59475f6f3fe375ec3]) successfully deleted
I20260924 04:41:16.267213 2629 tablet_bootstrap.cc:654] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:16.267254 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 34bbed415f9e4c5ebf7dc94e1db3469d (DEFAULT_TABLE table=test-table [id=1e070531d7584b8db4930c2e1492790c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:16.267555 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bbed415f9e4c5ebf7dc94e1db3469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:16.268148 2674 ts_tablet_manager.cc:1905] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:16.269248 2629 tablet_bootstrap.cc:492] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: No bootstrap required, opened a new log
I20260924 04:41:16.269343 2629 ts_tablet_manager.cc:1397] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 04:41:16.269850 2629 raft_consensus.cc:348] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.269995 2630 tablet_bootstrap.cc:492] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Bootstrap starting.
I20260924 04:41:16.270005 2629 raft_consensus.cc:374] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:16.270043 2629 raft_consensus.cc:729] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d868ba0d1f394806804df63e78f870ad, State: Initialized, Role: FOLLOWER
I20260924 04:41:16.270167 2629 consensus_queue.cc:260] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.270469 2629 ts_tablet_manager.cc:1428] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:41:16.270855 2630 tablet_bootstrap.cc:654] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:16.271692 2674 ts_tablet_manager.cc:1918] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:16.271845 2674 log.cc:1196] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/bc542dfff9644a8e912f5cee8afac072
I20260924 04:41:16.272171 2674 ts_tablet_manager.cc:1939] T bc542dfff9644a8e912f5cee8afac072 P 1595f623b10a40c982c16398b66507ee: Deleting consensus metadata
I20260924 04:41:16.273305 2630 tablet_bootstrap.cc:492] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: No bootstrap required, opened a new log
I20260924 04:41:16.273399 2630 ts_tablet_manager.cc:1397] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 04:41:16.273494 2631 tablet_bootstrap.cc:492] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: No bootstrap required, opened a new log
I20260924 04:41:16.273582 2631 ts_tablet_manager.cc:1397] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 04:41:16.273867 2630 raft_consensus.cc:348] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.274004 2630 raft_consensus.cc:374] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:16.273975 2631 raft_consensus.cc:348] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.274046 2630 raft_consensus.cc:729] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Initialized, Role: FOLLOWER
I20260924 04:41:16.274089 2631 raft_consensus.cc:374] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:16.274129 2631 raft_consensus.cc:729] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1595f623b10a40c982c16398b66507ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:16.274178 2630 consensus_queue.cc:260] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.274235 2631 consensus_queue.cc:260] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.274472 2630 ts_tablet_manager.cc:1428] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:41:16.274492 2631 ts_tablet_manager.cc:1428] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:16.275636 2107 catalog_manager.cc:4841] TS 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975): tablet bc542dfff9644a8e912f5cee8afac072 (table test-table [id=2edff08e81c545e59475f6f3fe375ec3]) successfully deleted
I20260924 04:41:16.469722 2636 raft_consensus.cc:482] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:16.469897 2636 raft_consensus.cc:504] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.471262 2636 leader_election.cc:288] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225)
I20260924 04:41:16.475449 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.471693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36775 (local address 127.0.98.2:46225)
0924 04:41:16.471837 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.471841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.472138 (+ 297us) server_negotiation.cc:407] Connection header received
0924 04:41:16.472260 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.472263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.472311 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.472474 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:16.472743 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.473252 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.474370 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.474781 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.474909 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.474982 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.475219 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.475304 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:41:16.475457 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.471692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52349 (local address 127.0.98.1:37813)
0924 04:41:16.472013 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.472032 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.472053 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:41:16.472116 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.472120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.472170 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.472257 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:16.472537 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.473147 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.474363 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.474781 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.474907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.474981 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.475227 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.475303 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":61,"threads_started":1}
I20260924 04:41:16.475471 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.471562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.98.3:52349)
0924 04:41:16.471928 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:16.471941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:16.472042 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:16.472278 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:16.472280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:16.472294 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:16.472415 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.472425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.473257 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.473260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.474217 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.474226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.474914 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.474917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.475074 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.475090 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:41:16.475208 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:41:16.475316 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":97,"thread_start_us":70,"threads_started":1}
I20260924 04:41:16.475482 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.471562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.98.3:36775)
0924 04:41:16.472009 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:16.472025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:16.472141 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:16.472502 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:16.472506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:16.472524 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:16.472633 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.472640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.473372 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.473375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.474221 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.474229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.474914 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.474917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.475043 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.475068 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:41:16.475200 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 04:41:16.475301 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":88,"thread_start_us":90,"threads_started":1}
I20260924 04:41:16.476075 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bbed415f9e4c5ebf7dc94e1db3469d" candidate_uuid: "1595f623b10a40c982c16398b66507ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad" is_pre_election: true
I20260924 04:41:16.476265 2280 raft_consensus.cc:2454] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1595f623b10a40c982c16398b66507ee in term 0.
I20260924 04:41:16.476274 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bbed415f9e4c5ebf7dc94e1db3469d" candidate_uuid: "1595f623b10a40c982c16398b66507ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c00f1753df84472a39817cbce7b7b5c" is_pre_election: true
I20260924 04:41:16.476519 2420 raft_consensus.cc:2454] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1595f623b10a40c982c16398b66507ee in term 0.
I20260924 04:41:16.476696 2494 leader_election.cc:302] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, d868ba0d1f394806804df63e78f870ad; no voters:
I20260924 04:41:16.477005 2636 raft_consensus.cc:2781] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:16.477075 2636 raft_consensus.cc:482] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:16.477113 2636 raft_consensus.cc:3037] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:16.489084 2636 raft_consensus.cc:504] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.489702 2636 leader_election.cc:288] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [CANDIDATE]: Term 1 election: Requested vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225)
I20260924 04:41:16.490171 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bbed415f9e4c5ebf7dc94e1db3469d" candidate_uuid: "1595f623b10a40c982c16398b66507ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad"
I20260924 04:41:16.490213 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bbed415f9e4c5ebf7dc94e1db3469d" candidate_uuid: "1595f623b10a40c982c16398b66507ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c00f1753df84472a39817cbce7b7b5c"
I20260924 04:41:16.490321 2280 raft_consensus.cc:3037] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:16.490362 2420 raft_consensus.cc:3037] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:16.491566 2280 raft_consensus.cc:2454] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1595f623b10a40c982c16398b66507ee in term 1.
I20260924 04:41:16.491961 2420 raft_consensus.cc:2454] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1595f623b10a40c982c16398b66507ee in term 1.
I20260924 04:41:16.491953 2494 leader_election.cc:302] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, d868ba0d1f394806804df63e78f870ad; no voters:
I20260924 04:41:16.492172 2636 raft_consensus.cc:2781] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:16.492388 2636 raft_consensus.cc:686] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 1 LEADER]: Becoming Leader. State: Replica: 1595f623b10a40c982c16398b66507ee, State: Running, Role: LEADER
I20260924 04:41:16.492662 2636 consensus_queue.cc:237] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:16.494939 2118 catalog_manager.cc:5495] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 1595f623b10a40c982c16398b66507ee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1595f623b10a40c982c16398b66507ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } health_report { overall_health: HEALTHY } } }
I20260924 04:41:16.516286 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.511996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.0.98.3:43975)
0924 04:41:16.512285 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.512296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.512315 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:41:16.512455 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.512458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.512525 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.512635 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:16.512846 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.513361 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.514434 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.514845 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.514956 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.515293 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.515856 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.516004 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.516064 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.516118 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260924 04:41:16.527161 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.523985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35523 (local address 127.0.98.2:46225)
0924 04:41:16.524124 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.524131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.524259 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:41:16.524378 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.524380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.524426 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.524492 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:16.524688 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.525184 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.526082 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.526331 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.526461 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.526720 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.526788 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.526842 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.526957 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.527004 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:16.527519 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.523905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.98.3:35523)
0924 04:41:16.524196 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:16.524209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:16.524301 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:16.524516 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:16.524518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:16.524525 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:16.524621 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.524625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.525280 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.525282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.525978 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.525986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.526476 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.526478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.526549 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.526590 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.526861 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.526867 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:41:16.526946 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:41:16.527013 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260924 04:41:16.533926 2280 raft_consensus.cc:1261] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Refusing update from remote peer 1595f623b10a40c982c16398b66507ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:16.534188 2420 raft_consensus.cc:1261] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Refusing update from remote peer 1595f623b10a40c982c16398b66507ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:16.534345 2636 consensus_queue.cc:1035] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:16.534699 2683 consensus_queue.cc:1035] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:16.553346 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.549501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43975 (local address 127.0.98.2:60823)
0924 04:41:16.549620 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:16.549636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:16.549710 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:16.550107 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:16.550113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:16.550126 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:16.550241 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.550251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.551044 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.551051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.552098 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:16.552109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.552581 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.552584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:16.552668 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.552715 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.553043 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.553054 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 04:41:16.553115 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:41:16.553206 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 04:41:16.553346 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:16.549576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60823 (local address 127.0.98.3:43975)
0924 04:41:16.549870 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:16.549875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:16.549888 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:41:16.549936 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:16.549938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:16.549990 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:16.550066 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:16.550367 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.550938 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.552226 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:16.552469 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:16.552584 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:16.552857 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:16.552938 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:16.553029 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:16.553139 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:41:16.553199 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
W20260924 04:41:16.554216 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:41:18.112119 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:18.112282 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:18.114807 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Sending DeleteTablet for 3 replicas of tablet 34bbed415f9e4c5ebf7dc94e1db3469d
I20260924 04:41:18.115417 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bbed415f9e4c5ebf7dc94e1db3469d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33438
I20260924 04:41:18.115491 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bbed415f9e4c5ebf7dc94e1db3469d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48940
I20260924 04:41:18.118701 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bbed415f9e4c5ebf7dc94e1db3469d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34350
I20260924 04:41:18.119093 2718 tablet_replica.cc:330] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: stopping tablet replica
I20260924 04:41:18.119314 2718 raft_consensus.cc:2229] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:18.119498 2718 raft_consensus.cc:2258] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:18.120527 2718 ts_tablet_manager.cc:1905] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:18.122879 2715 tablet_replica.cc:330] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: stopping tablet replica
I20260924 04:41:18.123014 2718 ts_tablet_manager.cc:1918] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:41:18.123060 2715 raft_consensus.cc:2229] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:18.123171 2718 log.cc:1196] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/34bbed415f9e4c5ebf7dc94e1db3469d
I20260924 04:41:18.123267 2715 raft_consensus.cc:2258] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:18.123489 2718 ts_tablet_manager.cc:1939] T 34bbed415f9e4c5ebf7dc94e1db3469d P 7c00f1753df84472a39817cbce7b7b5c: Deleting consensus metadata
I20260924 04:41:18.124308 2109 catalog_manager.cc:4841] TS 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225): tablet 34bbed415f9e4c5ebf7dc94e1db3469d (table test-table [id=1e070531d7584b8db4930c2e1492790c]) successfully deleted
I20260924 04:41:18.127347 2716 tablet_replica.cc:330] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: stopping tablet replica
I20260924 04:41:18.127535 2716 raft_consensus.cc:2229] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:18.127758 2715 ts_tablet_manager.cc:1905] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:18.127914 2716 raft_consensus.cc:2258] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:18.128546 2716 ts_tablet_manager.cc:1905] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 04:41:18 dist-test-slave-npg2 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260924 04:41:18.130429 2715 ts_tablet_manager.cc:1918] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:41:18.130524 2715 log.cc:1196] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/34bbed415f9e4c5ebf7dc94e1db3469d
I20260924 04:41:18.130832 2715 ts_tablet_manager.cc:1939] T 34bbed415f9e4c5ebf7dc94e1db3469d P d868ba0d1f394806804df63e78f870ad: Deleting consensus metadata
I20260924 04:41:18.131305 2716 ts_tablet_manager.cc:1918] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:41:18.131382 2716 log.cc:1196] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/34bbed415f9e4c5ebf7dc94e1db3469d
I20260924 04:41:18.131645 2716 ts_tablet_manager.cc:1939] T 34bbed415f9e4c5ebf7dc94e1db3469d P 1595f623b10a40c982c16398b66507ee: Deleting consensus metadata
I20260924 04:41:18.131652 2110 catalog_manager.cc:4841] TS d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813): tablet 34bbed415f9e4c5ebf7dc94e1db3469d (table test-table [id=1e070531d7584b8db4930c2e1492790c]) successfully deleted
I20260924 04:41:18.132318 2107 catalog_manager.cc:4841] TS 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975): tablet 34bbed415f9e4c5ebf7dc94e1db3469d (table test-table [id=1e070531d7584b8db4930c2e1492790c]) successfully deleted
Sep 24 04:41:18 dist-test-slave-npg2 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:18.153856 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.136609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.0.98.62:45549)
0924 04:41:18.136918 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.136927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.137035 (+ 108us) server_negotiation.cc:407] Connection header received
0924 04:41:18.137203 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.137207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.137260 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.137387 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:18.138070 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.138632 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.139543 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.139787 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.139900 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.150371 (+ 10471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:18.150374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:18.150376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:18.150406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:18.152054 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:18.152551 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:18.152553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:18.152554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:18.152626 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:18.152940 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:18.152947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:18.152948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:18.153127 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:18.153230 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.153581 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.153692 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260924 04:41:18.156515 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:18.157011 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:18.161665 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 957375cefeb0425880e95f9fa7d799b6 (DEFAULT_TABLE table=test-table [id=b45c32df6d074f21bdcbbc1ed0c49016]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:18.161701 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 957375cefeb0425880e95f9fa7d799b6 (DEFAULT_TABLE table=test-table [id=b45c32df6d074f21bdcbbc1ed0c49016]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:18.161783 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 957375cefeb0425880e95f9fa7d799b6 (DEFAULT_TABLE table=test-table [id=b45c32df6d074f21bdcbbc1ed0c49016]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:18.162010 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957375cefeb0425880e95f9fa7d799b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:18.162012 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957375cefeb0425880e95f9fa7d799b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:18.162036 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957375cefeb0425880e95f9fa7d799b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:18.164439 2726 tablet_bootstrap.cc:492] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Bootstrap starting.
I20260924 04:41:18.164891 2727 tablet_bootstrap.cc:492] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Bootstrap starting.
I20260924 04:41:18.165419 2726 tablet_bootstrap.cc:654] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:18.165786 2727 tablet_bootstrap.cc:654] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:18.166724 2726 tablet_bootstrap.cc:492] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: No bootstrap required, opened a new log
I20260924 04:41:18.166834 2726 ts_tablet_manager.cc:1397] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:18.166924 2728 tablet_bootstrap.cc:492] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Bootstrap starting.
I20260924 04:41:18.167328 2726 raft_consensus.cc:348] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.167447 2726 raft_consensus.cc:374] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:18.167491 2726 raft_consensus.cc:729] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1595f623b10a40c982c16398b66507ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:18.167622 2726 consensus_queue.cc:260] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.167752 2728 tablet_bootstrap.cc:654] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:18.167887 2726 ts_tablet_manager.cc:1428] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:18.168552 2727 tablet_bootstrap.cc:492] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: No bootstrap required, opened a new log
I20260924 04:41:18.168654 2727 ts_tablet_manager.cc:1397] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:41:18.169092 2727 raft_consensus.cc:348] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.169226 2727 raft_consensus.cc:374] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:18.169262 2727 raft_consensus.cc:729] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d868ba0d1f394806804df63e78f870ad, State: Initialized, Role: FOLLOWER
I20260924 04:41:18.169730 2728 tablet_bootstrap.cc:492] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: No bootstrap required, opened a new log
I20260924 04:41:18.169867 2728 ts_tablet_manager.cc:1397] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:18.170212 2728 raft_consensus.cc:348] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.170298 2728 raft_consensus.cc:374] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:18.170321 2728 raft_consensus.cc:729] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Initialized, Role: FOLLOWER
I20260924 04:41:18.170464 2728 consensus_queue.cc:260] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.170732 2728 ts_tablet_manager.cc:1428] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:41:18.170907 2727 consensus_queue.cc:260] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.171325 2727 ts_tablet_manager.cc:1428] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:18.366500 2732 raft_consensus.cc:482] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:18.366652 2732 raft_consensus.cc:504] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.367986 2732 leader_election.cc:288] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:18.373831 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.368377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60345 (local address 127.0.98.2:46225)
0924 04:41:18.368814 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.368824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.368844 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:41:18.368963 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.368965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.369019 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.369121 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:18.369623 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.370455 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.372483 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.373147 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.373265 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.373330 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.373606 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.373692 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":94,"threads_started":1}
I20260924 04:41:18.373831 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.368539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43999 (local address 127.0.98.3:43975)
0924 04:41:18.368814 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.368827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.368844 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:41:18.368962 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.368965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.369019 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.369121 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:18.369629 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.370465 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.372517 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.373138 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.373259 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.373330 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.373606 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.373692 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260924 04:41:18.373842 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.368324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43975 (local address 127.0.98.1:43999)
0924 04:41:18.368690 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:18.368708 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:18.368846 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:18.369187 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:18.369190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:18.369221 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:18.369411 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.369420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.370713 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.370717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.372329 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.372340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.373274 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.373278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.373445 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.373469 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:41:18.373538 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:41:18.373629 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":183,"spinlock_wait_cycles":28288,"thread_start_us":130,"threads_started":1}
I20260924 04:41:18.373845 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.368156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.98.1:60345)
0924 04:41:18.368690 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:18.368708 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:18.368846 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:18.369187 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:18.369190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:18.369223 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:18.369464 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.369475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.370713 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.370717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.372334 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.372340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.373274 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.373278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.373405 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.373433 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:41:18.373520 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:41:18.373629 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":134,"spinlock_wait_cycles":18944,"thread_start_us":151,"threads_started":1}
I20260924 04:41:18.374366 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957375cefeb0425880e95f9fa7d799b6" candidate_uuid: "d868ba0d1f394806804df63e78f870ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee" is_pre_election: true
I20260924 04:41:18.374544 2560 raft_consensus.cc:2454] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d868ba0d1f394806804df63e78f870ad in term 0.
I20260924 04:41:18.374756 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957375cefeb0425880e95f9fa7d799b6" candidate_uuid: "d868ba0d1f394806804df63e78f870ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c00f1753df84472a39817cbce7b7b5c" is_pre_election: true
I20260924 04:41:18.374909 2420 raft_consensus.cc:2454] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d868ba0d1f394806804df63e78f870ad in term 0.
I20260924 04:41:18.374974 2211 leader_election.cc:302] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, d868ba0d1f394806804df63e78f870ad; no voters:
I20260924 04:41:18.375278 2732 raft_consensus.cc:2781] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:18.375346 2732 raft_consensus.cc:482] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:18.375375 2732 raft_consensus.cc:3037] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:18.376536 2732 raft_consensus.cc:504] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.376953 2732 leader_election.cc:288] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [CANDIDATE]: Term 1 election: Requested vote from peers 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:18.377388 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957375cefeb0425880e95f9fa7d799b6" candidate_uuid: "d868ba0d1f394806804df63e78f870ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c00f1753df84472a39817cbce7b7b5c"
I20260924 04:41:18.377524 2420 raft_consensus.cc:3037] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:18.377487 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957375cefeb0425880e95f9fa7d799b6" candidate_uuid: "d868ba0d1f394806804df63e78f870ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee"
I20260924 04:41:18.377629 2560 raft_consensus.cc:3037] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:18.378539 2420 raft_consensus.cc:2454] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d868ba0d1f394806804df63e78f870ad in term 1.
I20260924 04:41:18.378638 2560 raft_consensus.cc:2454] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d868ba0d1f394806804df63e78f870ad in term 1.
I20260924 04:41:18.378858 2213 leader_election.cc:302] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c00f1753df84472a39817cbce7b7b5c, d868ba0d1f394806804df63e78f870ad; no voters:
I20260924 04:41:18.379045 2732 raft_consensus.cc:2781] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:18.379328 2732 raft_consensus.cc:686] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 1 LEADER]: Becoming Leader. State: Replica: d868ba0d1f394806804df63e78f870ad, State: Running, Role: LEADER
I20260924 04:41:18.379582 2732 consensus_queue.cc:237] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.381834 2118 catalog_manager.cc:5495] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad reported cstate change: term changed from 0 to 1, leader changed from <none> to d868ba0d1f394806804df63e78f870ad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d868ba0d1f394806804df63e78f870ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:18.412076 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.408365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.0.98.1:37813)
0924 04:41:18.408625 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.408632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.408644 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:18.408713 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.408715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.408765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.408867 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:18.409087 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.409620 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.410570 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.410803 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.410882 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.411117 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:18.411601 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:18.411689 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.411786 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.411833 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":122,"threads_started":1}
I20260924 04:41:18.423036 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.419865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39781 (local address 127.0.98.2:46225)
0924 04:41:18.420011 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.420017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.420031 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:18.420172 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.420175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.420228 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.420311 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:18.420560 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.421061 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.421962 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.422184 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.422300 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.422586 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:18.422661 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:18.422742 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.422836 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.422886 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:41:18.423053 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.419592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.98.1:39781)
0924 04:41:18.419937 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:18.419954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:18.420060 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:18.420323 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:18.420326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:18.420333 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:18.420456 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.420462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.421174 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.421177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.421831 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.421839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.422316 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.422319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.422408 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.422449 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:18.422747 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:18.422756 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:41:18.422816 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:41:18.422892 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":116,"threads_started":1}
I20260924 04:41:18.429733 2560 raft_consensus.cc:1261] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Refusing update from remote peer d868ba0d1f394806804df63e78f870ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:18.429785 2420 raft_consensus.cc:1261] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Refusing update from remote peer d868ba0d1f394806804df63e78f870ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:18.430235 2737 consensus_queue.cc:1035] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:18.430434 2732 consensus_queue.cc:1035] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:18.448822 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.444627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44797 (local address 127.0.98.1:37813)
0924 04:41:18.444742 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.444748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.444966 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:41:18.445073 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.445079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.445129 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.445230 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:18.445511 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.446188 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.447464 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.447765 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.447872 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.448238 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:18.448361 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:18.448450 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.448554 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.448673 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:41:18.448975 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.444570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.98.2:44797)
0924 04:41:18.444887 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:18.444901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:18.444993 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:18.445241 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:18.445244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:18.445252 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:18.445359 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.445365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.446332 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.446335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.447344 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:18.447354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.447955 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.447958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:18.448040 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.448071 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:18.448452 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:18.448461 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:41:18.448747 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 04:41:18.448831 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260924 04:41:18.472059 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:18.472208 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:18.474431 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Sending DeleteTablet for 3 replicas of tablet 957375cefeb0425880e95f9fa7d799b6
I20260924 04:41:18.475023 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 957375cefeb0425880e95f9fa7d799b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33438
I20260924 04:41:18.475097 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 957375cefeb0425880e95f9fa7d799b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34350
I20260924 04:41:18.475234 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 957375cefeb0425880e95f9fa7d799b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48940
I20260924 04:41:18.475342 2718 tablet_replica.cc:330] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: stopping tablet replica
I20260924 04:41:18.475442 2716 tablet_replica.cc:330] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: stopping tablet replica
I20260924 04:41:18.475494 2718 raft_consensus.cc:2229] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:18.475584 2716 raft_consensus.cc:2229] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:18.475626 2718 raft_consensus.cc:2258] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:18.475718 2716 raft_consensus.cc:2258] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:18.476086 2715 tablet_replica.cc:330] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: stopping tablet replica
I20260924 04:41:18.476224 2715 raft_consensus.cc:2229] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:18.476538 2715 raft_consensus.cc:2258] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:18.476612 2716 ts_tablet_manager.cc:1905] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:18.476699 2718 ts_tablet_manager.cc:1905] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:18.476830 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:18.477269 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:18.479774 2718 ts_tablet_manager.cc:1918] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:18.479774 2715 ts_tablet_manager.cc:1905] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:18.479852 2718 log.cc:1196] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/957375cefeb0425880e95f9fa7d799b6
I20260924 04:41:18.480122 2718 ts_tablet_manager.cc:1939] T 957375cefeb0425880e95f9fa7d799b6 P 7c00f1753df84472a39817cbce7b7b5c: Deleting consensus metadata
I20260924 04:41:18.480846 2109 catalog_manager.cc:4841] TS 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225): tablet 957375cefeb0425880e95f9fa7d799b6 (table test-table [id=b45c32df6d074f21bdcbbc1ed0c49016]) successfully deleted
I20260924 04:41:18.482546 2716 ts_tablet_manager.cc:1918] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:18.482630 2716 log.cc:1196] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/957375cefeb0425880e95f9fa7d799b6
I20260924 04:41:18.482764 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2f033ca3e24c16a5d0f363628bbe3e (DEFAULT_TABLE table=test-table [id=6ae0db0e79e54e389da2479ddcd59a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:18.482893 2716 ts_tablet_manager.cc:1939] T 957375cefeb0425880e95f9fa7d799b6 P 1595f623b10a40c982c16398b66507ee: Deleting consensus metadata
I20260924 04:41:18.483018 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2f033ca3e24c16a5d0f363628bbe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:18.483466 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2f033ca3e24c16a5d0f363628bbe3e (DEFAULT_TABLE table=test-table [id=6ae0db0e79e54e389da2479ddcd59a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:18.483698 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2f033ca3e24c16a5d0f363628bbe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:18.484392 2107 catalog_manager.cc:4841] TS 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975): tablet 957375cefeb0425880e95f9fa7d799b6 (table test-table [id=b45c32df6d074f21bdcbbc1ed0c49016]) successfully deleted
I20260924 04:41:18.484721 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2f033ca3e24c16a5d0f363628bbe3e (DEFAULT_TABLE table=test-table [id=6ae0db0e79e54e389da2479ddcd59a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:18.484969 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2f033ca3e24c16a5d0f363628bbe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:18.485324 2715 ts_tablet_manager.cc:1918] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:18.485396 2715 log.cc:1196] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/957375cefeb0425880e95f9fa7d799b6
I20260924 04:41:18.485663 2715 ts_tablet_manager.cc:1939] T 957375cefeb0425880e95f9fa7d799b6 P d868ba0d1f394806804df63e78f870ad: Deleting consensus metadata
I20260924 04:41:18.486004 2727 tablet_bootstrap.cc:492] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad: Bootstrap starting.
I20260924 04:41:18.486231 2110 catalog_manager.cc:4841] TS d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813): tablet 957375cefeb0425880e95f9fa7d799b6 (table test-table [id=b45c32df6d074f21bdcbbc1ed0c49016]) successfully deleted
I20260924 04:41:18.486941 2727 tablet_bootstrap.cc:654] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:18.487317 2726 tablet_bootstrap.cc:492] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Bootstrap starting.
I20260924 04:41:18.487452 2728 tablet_bootstrap.cc:492] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Bootstrap starting.
I20260924 04:41:18.488139 2726 tablet_bootstrap.cc:654] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:18.488139 2728 tablet_bootstrap.cc:654] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:18.489487 2728 tablet_bootstrap.cc:492] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: No bootstrap required, opened a new log
I20260924 04:41:18.489494 2726 tablet_bootstrap.cc:492] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: No bootstrap required, opened a new log
I20260924 04:41:18.489566 2728 ts_tablet_manager.cc:1397] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:18.489573 2726 ts_tablet_manager.cc:1397] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:18.489938 2728 raft_consensus.cc:348] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.490051 2728 raft_consensus.cc:374] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:18.490087 2728 raft_consensus.cc:729] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Initialized, Role: FOLLOWER
I20260924 04:41:18.490021 2726 raft_consensus.cc:348] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.490130 2726 raft_consensus.cc:374] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:18.490168 2726 raft_consensus.cc:729] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1595f623b10a40c982c16398b66507ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:18.490198 2728 consensus_queue.cc:260] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.490307 2726 consensus_queue.cc:260] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.490459 2728 ts_tablet_manager.cc:1428] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:18.490620 2726 ts_tablet_manager.cc:1428] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:41:18.491582 2727 tablet_bootstrap.cc:492] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad: No bootstrap required, opened a new log
I20260924 04:41:18.491678 2727 ts_tablet_manager.cc:1397] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260924 04:41:18.492146 2727 raft_consensus.cc:348] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.492271 2727 raft_consensus.cc:374] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:18.492305 2727 raft_consensus.cc:729] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d868ba0d1f394806804df63e78f870ad, State: Initialized, Role: FOLLOWER
I20260924 04:41:18.492429 2727 consensus_queue.cc:260] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.492683 2727 ts_tablet_manager.cc:1428] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:41:18.612430 2749 raft_consensus.cc:482] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:18.612613 2749 raft_consensus.cc:504] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.613094 2749 leader_election.cc:288] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:18.613543 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2f033ca3e24c16a5d0f363628bbe3e" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee" is_pre_election: true
I20260924 04:41:18.613554 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2f033ca3e24c16a5d0f363628bbe3e" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad" is_pre_election: true
I20260924 04:41:18.613708 2280 raft_consensus.cc:2454] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 0.
I20260924 04:41:18.613708 2560 raft_consensus.cc:2454] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 0.
I20260924 04:41:18.614032 2354 leader_election.cc:302] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c00f1753df84472a39817cbce7b7b5c, d868ba0d1f394806804df63e78f870ad; no voters:
I20260924 04:41:18.614212 2749 raft_consensus.cc:2781] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:18.614288 2749 raft_consensus.cc:482] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:18.614342 2749 raft_consensus.cc:3037] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:18.615507 2749 raft_consensus.cc:504] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.615890 2749 leader_election.cc:288] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 election: Requested vote from peers d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813), 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975)
I20260924 04:41:18.616220 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2f033ca3e24c16a5d0f363628bbe3e" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d868ba0d1f394806804df63e78f870ad"
I20260924 04:41:18.616238 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2f033ca3e24c16a5d0f363628bbe3e" candidate_uuid: "7c00f1753df84472a39817cbce7b7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595f623b10a40c982c16398b66507ee"
I20260924 04:41:18.616353 2280 raft_consensus.cc:3037] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:18.616376 2560 raft_consensus.cc:3037] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:18.617367 2560 raft_consensus.cc:2454] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 1.
I20260924 04:41:18.617563 2280 raft_consensus.cc:2454] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c00f1753df84472a39817cbce7b7b5c in term 1.
I20260924 04:41:18.617730 2351 leader_election.cc:302] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595f623b10a40c982c16398b66507ee, 7c00f1753df84472a39817cbce7b7b5c; no voters:
I20260924 04:41:18.617899 2749 raft_consensus.cc:2781] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:18.618016 2749 raft_consensus.cc:686] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 1 LEADER]: Becoming Leader. State: Replica: 7c00f1753df84472a39817cbce7b7b5c, State: Running, Role: LEADER
I20260924 04:41:18.618156 2749 consensus_queue.cc:237] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } }
I20260924 04:41:18.619990 2122 catalog_manager.cc:5495] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7c00f1753df84472a39817cbce7b7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c00f1753df84472a39817cbce7b7b5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:18.649567 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:18.646242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.0.98.2:46225)
0924 04:41:18.646376 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:18.646381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:18.646529 (+ 148us) server_negotiation.cc:407] Connection header received
0924 04:41:18.646676 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:18.646679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:18.646733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:18.646818 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:18.647065 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.647613 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.648457 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:18.648751 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:18.648878 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:18.649126 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:18.649208 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:18.649289 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:18.649379 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:41:18.649427 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:41:18.655553 2560 raft_consensus.cc:1261] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Refusing update from remote peer 7c00f1753df84472a39817cbce7b7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:18.655663 2280 raft_consensus.cc:1261] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad [term 1 FOLLOWER]: Refusing update from remote peer 7c00f1753df84472a39817cbce7b7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:18.656093 2749 consensus_queue.cc:1035] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1595f623b10a40c982c16398b66507ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:18.657146 2749 consensus_queue.cc:1035] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d868ba0d1f394806804df63e78f870ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:41:18.669751 2694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:41:20.220655 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:20.220808 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:20.223237 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7188e74b58847eeb5a329c95ee1f8c0: Sending DeleteTablet for 3 replicas of tablet 7d2f033ca3e24c16a5d0f363628bbe3e
I20260924 04:41:20.223847 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2f033ca3e24c16a5d0f363628bbe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34350
I20260924 04:41:20.223857 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2f033ca3e24c16a5d0f363628bbe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33438
I20260924 04:41:20.223990 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2f033ca3e24c16a5d0f363628bbe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48940
I20260924 04:41:20.224234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 2195
I20260924 04:41:20.224277 2778 tablet_replica.cc:330] T 7d2f033ca3e24c16a5d0f363628bbe3e P d868ba0d1f394806804df63e78f870ad: stopping tablet replica
I20260924 04:41:20.225543 2777 tablet_replica.cc:330] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: stopping tablet replica
I20260924 04:41:20.225700 2777 raft_consensus.cc:2229] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:20.225931 2777 raft_consensus.cc:2258] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:20.226302 2779 tablet_replica.cc:330] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: stopping tablet replica
I20260924 04:41:20.226468 2779 raft_consensus.cc:2229] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:20.226611 2779 raft_consensus.cc:2258] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:20.226752 2777 ts_tablet_manager.cc:1905] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:20.227263 2779 ts_tablet_manager.cc:1905] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:20.229281 2779 ts_tablet_manager.cc:1918] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:41:20.229377 2779 log.cc:1196] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/7d2f033ca3e24c16a5d0f363628bbe3e
I20260924 04:41:20.229645 2779 ts_tablet_manager.cc:1939] T 7d2f033ca3e24c16a5d0f363628bbe3e P 1595f623b10a40c982c16398b66507ee: Deleting consensus metadata
I20260924 04:41:20.230252 2777 ts_tablet_manager.cc:1918] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:41:20.230340 2777 log.cc:1196] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/7d2f033ca3e24c16a5d0f363628bbe3e
I20260924 04:41:20.230338 2107 catalog_manager.cc:4841] TS 1595f623b10a40c982c16398b66507ee (127.0.98.3:43975): tablet 7d2f033ca3e24c16a5d0f363628bbe3e (table test-table [id=6ae0db0e79e54e389da2479ddcd59a1f]) successfully deleted
I20260924 04:41:20.230639 2777 ts_tablet_manager.cc:1939] T 7d2f033ca3e24c16a5d0f363628bbe3e P 7c00f1753df84472a39817cbce7b7b5c: Deleting consensus metadata
I20260924 04:41:20.231410 2109 catalog_manager.cc:4841] TS 7c00f1753df84472a39817cbce7b7b5c (127.0.98.2:46225): tablet 7d2f033ca3e24c16a5d0f363628bbe3e (table test-table [id=6ae0db0e79e54e389da2479ddcd59a1f]) successfully deleted
W20260924 04:41:20.233157 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37813 (error 108)
I20260924 04:41:20.233898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 2335
I20260924 04:41:20.234938 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:20.233883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.0.1:33458)
0924 04:41:20.234750 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:20.234834 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":103,"threads_started":1}
W20260924 04:41:20.235105 2110 catalog_manager.cc:4584] TS d868ba0d1f394806804df63e78f870ad (127.0.98.1:37813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d2f033ca3e24c16a5d0f363628bbe3e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37813: connect: Connection refused (error 111)
I20260924 04:41:20.241727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 2475
I20260924 04:41:20.248167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 2091
2026-09-24T04:41:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6838 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:20 dist-test-slave-npg2 krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:20 dist-test-slave-npg2 krb5kdc[2784](info): set up 2 sockets
Sep 24 04:41:20 dist-test-slave-npg2 krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260924 04:41:21.299194 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:21 dist-test-slave-npg2 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:21.426559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:21.539635 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:21.539909 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:21.539951 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:21.543967 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:21.544023 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:21.544042 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:21.544059 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:21.544075 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:21.549373 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:21.550372 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:21.551347 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:21.559942 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:21.560045 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:21.559962 2797 server_base.cc:1034] running on GCE node
W20260924 04:41:21.560034 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:21.560753 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:21.561877 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:21.563071 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790224881563043 us; error 43 us; skew 500 ppm
Sep 24 04:41:21 dist-test-slave-npg2 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:21.565739 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:21.566617 2797 webserver.cc:460] Webserver started at http://127.0.98.62:39521/ using document root <none> and password file <none>
I20260924 04:41:21.567207 2797 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:21.567271 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:21.567440 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:21.569387 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "0255bcc9aedf48dbad998e1d4ccf030a"
format_stamp: "Formatted at 2026-09-24 04:41:21 on dist-test-slave-npg2"
server_key: "e04adf6c72a50c6ed7dd0e2e8f6e28c7"
server_key_iv: "1d628d379ba906b6dc100ad6bb826d71"
server_key_version: "encryptionkey@0"
I20260924 04:41:21.569882 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "0255bcc9aedf48dbad998e1d4ccf030a"
format_stamp: "Formatted at 2026-09-24 04:41:21 on dist-test-slave-npg2"
server_key: "e04adf6c72a50c6ed7dd0e2e8f6e28c7"
server_key_iv: "1d628d379ba906b6dc100ad6bb826d71"
server_key_version: "encryptionkey@0"
I20260924 04:41:21.573451 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 04:41:21.575870 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:21.577204 2797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:41:21.577314 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "0255bcc9aedf48dbad998e1d4ccf030a"
format_stamp: "Formatted at 2026-09-24 04:41:21 on dist-test-slave-npg2"
server_key: "e04adf6c72a50c6ed7dd0e2e8f6e28c7"
server_key_iv: "1d628d379ba906b6dc100ad6bb826d71"
server_key_version: "encryptionkey@0"
I20260924 04:41:21.577481 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:21.603523 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:21.607189 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:21.607359 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:21.614532 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39197
I20260924 04:41:21.614557 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39197 every 8 connection(s)
I20260924 04:41:21.615542 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:21.618487 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:21.623157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 2797
I20260924 04:41:21.623265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:21.623549 392 external_mini_cluster.cc:1428] Setting key ca60f546588f2644fdf72404a54402ed
I20260924 04:41:21.624662 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a: Bootstrap starting.
I20260924 04:41:21.627002 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:21.627812 2865 log.cc:824] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:21.629622 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a: No bootstrap required, opened a new log
Sep 24 04:41:21 dist-test-slave-npg2 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:21.633553 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260924 04:41:21.633802 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:21.633869 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0255bcc9aedf48dbad998e1d4ccf030a, State: Initialized, Role: FOLLOWER
I20260924 04:41:21.634349 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260924 04:41:21.634505 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:21.634581 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:21.634696 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:21.635638 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260924 04:41:21.635968 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0255bcc9aedf48dbad998e1d4ccf030a; no voters:
I20260924 04:41:21.636255 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:21.636363 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:21.636633 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [term 1 LEADER]: Becoming Leader. State: Replica: 0255bcc9aedf48dbad998e1d4ccf030a, State: Running, Role: LEADER
I20260924 04:41:21.637169 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260924 04:41:21.637558 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:21.638007 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:21.625063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.0.98.62:39197)
0924 04:41:21.625515 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:21.625530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:21.625566 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:21.626213 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:21.626230 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:21.626570 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:21.626911 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:21.627724 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:21.628768 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:21.629644 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:21.630165 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:21.630339 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:21.632903 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:21.632908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:21.632922 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:21.632948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:21.634833 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:21.635323 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:21.635329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:21.635334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:21.635462 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:21.635742 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:21.635746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:21.635750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:21.636206 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:21.636416 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:21.636729 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 04:41:21.637048 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":132,"threads_started":1}
I20260924 04:41:21.638371 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } } }
I20260924 04:41:21.638486 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:21.638669 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0255bcc9aedf48dbad998e1d4ccf030a. Latest consensus state: current_term: 1 leader_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255bcc9aedf48dbad998e1d4ccf030a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } } }
I20260924 04:41:21.638744 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:21.638988 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:21.642462 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:21.648094 2879 catalog_manager.cc:1347] Generated new cluster ID: a76e98e8695243d787cb28eb1b2c183a
I20260924 04:41:21.648164 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:21.659731 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:21.660638 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:21.669920 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0255bcc9aedf48dbad998e1d4ccf030a: Generated new TSK 0
I20260924 04:41:21.670616 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:21.702545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39197
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:21.815465 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:21.815735 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:21.815778 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:21.819914 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:21.819983 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:21.820042 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:21.825367 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:21.826442 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:21.827526 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:21.837554 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:21.837554 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:21.837610 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:21.838083 2891 server_base.cc:1034] running on GCE node
I20260924 04:41:21.838476 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:21.839188 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:21.840363 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790224881840346 us; error 34 us; skew 500 ppm
Sep 24 04:41:21 dist-test-slave-npg2 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:21.843219 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:21.844133 2891 webserver.cc:460] Webserver started at http://127.0.98.1:45223/ using document root <none> and password file <none>
I20260924 04:41:21.844719 2891 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:21.844781 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:21.844962 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:21.846989 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "8419626d4fe44e43b7d3a6988ae5cd5a"
format_stamp: "Formatted at 2026-09-24 04:41:21 on dist-test-slave-npg2"
server_key: "08ec792a52cb95ffdaba2a41c3ee3699"
server_key_iv: "2267eec5901299c3c1e587ed77879f54"
server_key_version: "encryptionkey@0"
I20260924 04:41:21.847563 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8419626d4fe44e43b7d3a6988ae5cd5a"
format_stamp: "Formatted at 2026-09-24 04:41:21 on dist-test-slave-npg2"
server_key: "08ec792a52cb95ffdaba2a41c3ee3699"
server_key_iv: "2267eec5901299c3c1e587ed77879f54"
server_key_version: "encryptionkey@0"
I20260924 04:41:21.851296 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:21.853713 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:21.854975 2891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:21.855113 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "8419626d4fe44e43b7d3a6988ae5cd5a"
format_stamp: "Formatted at 2026-09-24 04:41:21 on dist-test-slave-npg2"
server_key: "08ec792a52cb95ffdaba2a41c3ee3699"
server_key_iv: "2267eec5901299c3c1e587ed77879f54"
server_key_version: "encryptionkey@0"
I20260924 04:41:21.855250 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:21.886013 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:21.889420 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:21.889637 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:21.890288 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:21.891158 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:21.891206 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:21.891244 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:21.891294 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:21.901010 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33831
I20260924 04:41:21.901031 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33831 every 8 connection(s)
I20260924 04:41:21.902132 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:21.909343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 2891
I20260924 04:41:21.909442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:21.909718 392 external_mini_cluster.cc:1428] Setting key 22c6530078e1bfd5f090006be9c41cb3
Sep 24 04:41:21 dist-test-slave-npg2 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:21.917086 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:21.904142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33209 (local address 127.0.98.62:39197)
0924 04:41:21.904314 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:21.904318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:21.905357 (+ 1039us) server_negotiation.cc:407] Connection header received
0924 04:41:21.906803 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:21.906806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:21.906864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:21.906972 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:21.908316 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:21.908847 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:21.910194 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:21.910535 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:21.910627 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:21.913516 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:21.913519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:21.913522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:21.913546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:21.915000 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:21.915551 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:21.915556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:21.915558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:21.915640 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:21.915981 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:21.915985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:21.915986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:21.916171 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:21.916272 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:21.916795 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 04:41:21.916923 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 04:41:21.917948 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:21.904696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39197 (local address 127.0.98.1:33209)
0924 04:41:21.905189 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:21.905234 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:21.906568 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:21.907173 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:21.907180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:21.907595 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:21.908160 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:21.908175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:21.908982 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:21.908986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:21.910031 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:21.910056 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:21.910690 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:21.910694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:21.910886 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:21.911618 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:21.911634 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:21.913329 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:21.915155 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:21.915163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:21.915181 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:21.915448 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:21.915737 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:21.915740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:21.915742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:21.915882 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:21.916283 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:21.916291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:21.916554 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 04:41:21.916741 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 04:41:21.917093 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:21.919462 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39197
I20260924 04:41:21.919682 3020 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:21.920194 3020 heartbeater.cc:511] Master 127.0.98.62:39197 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:21.921749 2827 ts_manager.cc:194] Registered new tserver with Master: 8419626d4fe44e43b7d3a6988ae5cd5a (127.0.98.1:33831)
I20260924 04:41:21.923377 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 04:41:22.043578 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.132s user 0.000s sys 0.003s
I20260924 04:41:22.044145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39197
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:22.157119 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:22.157411 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:22.157459 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:22.162016 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:22.162088 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:22.162151 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:22.167683 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:22.168711 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:22.169859 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:22.179589 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:22.179562 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:22.179641 3025 server_base.cc:1034] running on GCE node
W20260924 04:41:22.179891 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:22.180305 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:22.180975 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:22.182164 3025 hybrid_clock.cc:648] HybridClock initialized: now 1790224882182095 us; error 83 us; skew 500 ppm
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:22.185165 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:22.186065 3025 webserver.cc:460] Webserver started at http://127.0.98.2:40321/ using document root <none> and password file <none>
I20260924 04:41:22.186630 3025 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:22.186692 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:22.186951 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:22.189061 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "119ae57aba3b4f1baa2cfe9cc3e07ffc"
format_stamp: "Formatted at 2026-09-24 04:41:22 on dist-test-slave-npg2"
server_key: "389b87c09b567f210fddc93fdeed12a9"
server_key_iv: "197cf8733dbe44da32e8ab7e73b23ee5"
server_key_version: "encryptionkey@0"
I20260924 04:41:22.189623 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "119ae57aba3b4f1baa2cfe9cc3e07ffc"
format_stamp: "Formatted at 2026-09-24 04:41:22 on dist-test-slave-npg2"
server_key: "389b87c09b567f210fddc93fdeed12a9"
server_key_iv: "197cf8733dbe44da32e8ab7e73b23ee5"
server_key_version: "encryptionkey@0"
I20260924 04:41:22.193485 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 04:41:22.196197 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:22.197394 3025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:41:22.197535 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "119ae57aba3b4f1baa2cfe9cc3e07ffc"
format_stamp: "Formatted at 2026-09-24 04:41:22 on dist-test-slave-npg2"
server_key: "389b87c09b567f210fddc93fdeed12a9"
server_key_iv: "197cf8733dbe44da32e8ab7e73b23ee5"
server_key_version: "encryptionkey@0"
I20260924 04:41:22.197643 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:22.215688 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:22.218904 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:22.219213 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:22.219866 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:22.220742 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:22.220789 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:22.220831 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:22.220846 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:22.230370 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35433
I20260924 04:41:22.230396 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35433 every 8 connection(s)
I20260924 04:41:22.231375 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:22.240653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3025
I20260924 04:41:22.240772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:22.241065 392 external_mini_cluster.cc:1428] Setting key 12b1adeab17c550b25f7e315f4c73883
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:22.247640 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:22.233249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37537 (local address 127.0.98.62:39197)
0924 04:41:22.233379 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:22.233388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:22.234344 (+ 956us) server_negotiation.cc:407] Connection header received
0924 04:41:22.235862 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:22.235866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:22.235926 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:22.236010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:22.237319 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.237935 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.239356 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.239715 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.239829 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:22.242861 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:22.242864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:22.242866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:22.242889 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:22.244807 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:22.245565 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:22.245568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:22.245569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:22.245697 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:22.246187 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:22.246193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:22.246198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:22.246482 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:22.246591 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:22.247179 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 04:41:22.247345 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:41:22.248478 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:22.233698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39197 (local address 127.0.98.2:37537)
0924 04:41:22.234190 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:22.234236 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:22.235632 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:22.236190 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:22.236198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:22.236610 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:22.237161 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:22.237176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.238086 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.238089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:22.239231 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:22.239242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.239881 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.239885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:22.240078 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:22.240842 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:22.240866 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:22.242683 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:22.244955 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:22.244965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:22.244988 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:22.245413 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:22.245923 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:22.245925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:22.245927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:22.246083 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:22.246593 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:22.246599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:22.246889 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 04:41:22.247066 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 04:41:22.247511 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":116,"threads_started":1}
I20260924 04:41:22.249760 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39197
I20260924 04:41:22.250026 3154 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:22.250561 3154 heartbeater.cc:511] Master 127.0.98.62:39197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:41:22.251714 2827 ts_manager.cc:194] Registered new tserver with Master: 119ae57aba3b4f1baa2cfe9cc3e07ffc (127.0.98.2:35433)
I20260924 04:41:22.252466 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:22.270179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39197
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:22.385195 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:22.385542 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:22.385591 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:22.389719 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:22.389788 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:22.389849 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:22.395335 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:22.396481 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:22.397516 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:22.407759 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:22.407822 3159 server_base.cc:1034] running on GCE node
W20260924 04:41:22.407759 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:22.407797 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:22.408499 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:22.409204 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:22.410393 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790224882410372 us; error 32 us; skew 500 ppm
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:22.413523 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:22.414532 3159 webserver.cc:460] Webserver started at http://127.0.98.3:35463/ using document root <none> and password file <none>
I20260924 04:41:22.415184 3159 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:22.415274 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:22.415512 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:22.417730 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "656e9ec5b0b94be1b81c9cba43580755"
format_stamp: "Formatted at 2026-09-24 04:41:22 on dist-test-slave-npg2"
server_key: "f792d72b0e63d57fab7181f0116b55f0"
server_key_iv: "f1f32d79eaaae965cea8f79c3d9da876"
server_key_version: "encryptionkey@0"
I20260924 04:41:22.418350 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "656e9ec5b0b94be1b81c9cba43580755"
format_stamp: "Formatted at 2026-09-24 04:41:22 on dist-test-slave-npg2"
server_key: "f792d72b0e63d57fab7181f0116b55f0"
server_key_iv: "f1f32d79eaaae965cea8f79c3d9da876"
server_key_version: "encryptionkey@0"
I20260924 04:41:22.422333 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:41:22.425033 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:22.426645 3159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:22.426805 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "656e9ec5b0b94be1b81c9cba43580755"
format_stamp: "Formatted at 2026-09-24 04:41:22 on dist-test-slave-npg2"
server_key: "f792d72b0e63d57fab7181f0116b55f0"
server_key_iv: "f1f32d79eaaae965cea8f79c3d9da876"
server_key_version: "encryptionkey@0"
I20260924 04:41:22.426972 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:22.444298 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:22.447762 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:22.447978 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:22.448582 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:22.449429 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:22.449496 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:22.449563 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:22.449607 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:22.458794 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36251
I20260924 04:41:22.458812 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36251 every 8 connection(s)
I20260924 04:41:22.459918 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:22.466845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3159
I20260924 04:41:22.466974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:22.467305 392 external_mini_cluster.cc:1428] Setting key ddb8fd012449ff55815babda3b417fda
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:22.475201 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:22.461895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51869 (local address 127.0.98.62:39197)
0924 04:41:22.462089 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:22.462094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:22.463262 (+ 1168us) server_negotiation.cc:407] Connection header received
0924 04:41:22.465097 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:22.465101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:22.465160 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:22.465241 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:22.466473 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.467080 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.468598 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.468965 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.469062 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:22.471703 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:22.471706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:22.471708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:22.471733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:22.473305 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:22.473803 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:22.473805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:22.473807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:22.473881 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:22.474165 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:22.474168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:22.474169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:22.474337 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:22.474412 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:22.474901 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 04:41:22.475023 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 04:41:22.476095 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:22.462426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39197 (local address 127.0.98.3:51869)
0924 04:41:22.463052 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:22.463114 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:22.464871 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:22.465416 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:22.465425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:22.465780 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:22.466280 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:22.466293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.467229 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.467233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:22.468474 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:22.468483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:22.469124 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:22.469128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:22.469330 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:22.469996 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:22.470012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:22.471530 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:22.473419 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:22.473425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:22.473439 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:22.473717 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:22.473960 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:22.473963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:22.473964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:22.474092 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:22.474420 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:22.474426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:22.474689 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 04:41:22.474853 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:41:22.475238 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":148,"threads_started":1}
I20260924 04:41:22.477236 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39197
I20260924 04:41:22.477461 3288 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:22.477928 3288 heartbeater.cc:511] Master 127.0.98.62:39197 requested a full tablet report, sending...
I20260924 04:41:22.478906 2827 ts_manager.cc:194] Registered new tserver with Master: 656e9ec5b0b94be1b81c9cba43580755 (127.0.98.3:36251)
I20260924 04:41:22.479516 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51869
I20260924 04:41:22.481367 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:41:22.494441 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:22.492960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.0.98.62:39197)
0924 04:41:22.493127 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:22.493132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:22.493281 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:41:22.493455 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:22.493457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:22.493502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:22.493586 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:22.494324 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58918: BlockingRecv error: recv got EOF from 127.0.0.1:58918 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:22.495921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 2891
I20260924 04:41:22.502326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3025
I20260924 04:41:22.508083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3159
I20260924 04:41:22.513659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 2797
2026-09-24T04:41:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2264 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[3300](info): set up 2 sockets
Sep 24 04:41:22 dist-test-slave-npg2 krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260924 04:41:23.583377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:23 dist-test-slave-npg2 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:23.710834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:23.823110 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:23.823413 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:23.823453 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:23.827577 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:23.827634 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:23.827653 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:23.827670 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:23.827685 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:23.833178 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:23.834303 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:23.835407 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:23.844276 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:23.844285 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:23.844478 3313 server_base.cc:1034] running on GCE node
W20260924 04:41:23.844322 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:23.845413 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:23.846630 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:23.847797 3313 hybrid_clock.cc:648] HybridClock initialized: now 1790224883847740 us; error 72 us; skew 500 ppm
Sep 24 04:41:23 dist-test-slave-npg2 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:23.860743 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:23.861795 3313 webserver.cc:460] Webserver started at http://127.0.98.62:42619/ using document root <none> and password file <none>
I20260924 04:41:23.862406 3313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:23.862496 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:23.862730 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:23.864929 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "9f0a46742b4f4a7d9711b0955b651bb6"
format_stamp: "Formatted at 2026-09-24 04:41:23 on dist-test-slave-npg2"
server_key: "d0d691071e4be26715e15fe5aedd2866"
server_key_iv: "ddaafd498b069ac08a2df7bfb01ff318"
server_key_version: "encryptionkey@0"
I20260924 04:41:23.865551 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f0a46742b4f4a7d9711b0955b651bb6"
format_stamp: "Formatted at 2026-09-24 04:41:23 on dist-test-slave-npg2"
server_key: "d0d691071e4be26715e15fe5aedd2866"
server_key_iv: "ddaafd498b069ac08a2df7bfb01ff318"
server_key_version: "encryptionkey@0"
I20260924 04:41:23.869313 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:41:23.871840 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:23.873131 3313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:23.873230 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "9f0a46742b4f4a7d9711b0955b651bb6"
format_stamp: "Formatted at 2026-09-24 04:41:23 on dist-test-slave-npg2"
server_key: "d0d691071e4be26715e15fe5aedd2866"
server_key_iv: "ddaafd498b069ac08a2df7bfb01ff318"
server_key_version: "encryptionkey@0"
I20260924 04:41:23.873319 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:23.900926 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:23.904608 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:23.904775 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:23.911808 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33669
I20260924 04:41:23.912109 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33669 every 8 connection(s)
I20260924 04:41:23.912846 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:23.916079 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:23.918032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3313
I20260924 04:41:23.918133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:23.918407 392 external_mini_cluster.cc:1428] Setting key fafcbb2d3461c84d3fcb75cf84f7024c
I20260924 04:41:23.922765 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6: Bootstrap starting.
I20260924 04:41:23.925298 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:23.926050 3381 log.cc:824] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:41:23 dist-test-slave-npg2 krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:23.928296 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6: No bootstrap required, opened a new log
I20260924 04:41:23.933182 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:23.919825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.0.98.62:33669)
0924 04:41:23.920297 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:23.920312 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:23.920349 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:41:23.921027 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:23.921044 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:23.921373 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:23.921721 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:23.922528 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:23.923617 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:23.924467 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:23.924942 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:23.925120 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:23.927733 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:23.927739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:23.927755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:23.927785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:23.929763 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:23.930278 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:23.930284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:23.930290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:23.930497 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:23.930791 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:23.930795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:23.930798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:23.931406 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:23.931594 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:23.931845 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:41:23.932166 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":163,"threads_started":1}
I20260924 04:41:23.933228 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33669 } }
I20260924 04:41:23.933490 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:23.933555 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0a46742b4f4a7d9711b0955b651bb6, State: Initialized, Role: FOLLOWER
I20260924 04:41:23.934031 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33669 } }
I20260924 04:41:23.934186 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:23.934296 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:23.934398 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:23.935559 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33669 } }
I20260924 04:41:23.935930 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f0a46742b4f4a7d9711b0955b651bb6; no voters:
I20260924 04:41:23.936272 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:23.936412 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:41:23.936474 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:23.936657 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [term 1 LEADER]: Becoming Leader. State: Replica: 9f0a46742b4f4a7d9711b0955b651bb6, State: Running, Role: LEADER
I20260924 04:41:23.937029 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33669 } }
I20260924 04:41:23.937668 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:23.938376 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f0a46742b4f4a7d9711b0955b651bb6. Latest consensus state: current_term: 1 leader_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33669 } } }
I20260924 04:41:23.938611 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:23.938810 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0a46742b4f4a7d9711b0955b651bb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33669 } } }
I20260924 04:41:23.938930 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:23.939057 3396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:23.942196 3396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:23.951210 3396 catalog_manager.cc:1347] Generated new cluster ID: 33ca6cb65d6948feb74392f31d0cb43d
I20260924 04:41:23.951287 3396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:23.963366 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:23.957719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.0.98.62:33669)
0924 04:41:23.957843 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:23.957847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:23.958045 (+ 198us) server_negotiation.cc:407] Connection header received
0924 04:41:23.958160 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:23.958163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:23.958211 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:23.958324 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:23.958880 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:23.959711 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:23.960807 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:23.961177 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:23.961283 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:23.962213 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:23.962217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:23.962220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:23.962837 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:23.962873 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:23.962893 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:23.962988 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:23.963188 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 04:41:23.963245 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:41:23.964102 3396 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:23.965327 3396 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:23.988581 3396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f0a46742b4f4a7d9711b0955b651bb6: Generated new TSK 0
I20260924 04:41:23.989506 3396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:41:24.000777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3313
2026-09-24T04:41:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1485 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:24 dist-test-slave-npg2 krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:24 dist-test-slave-npg2 krb5kdc[3414](info): set up 2 sockets
Sep 24 04:41:24 dist-test-slave-npg2 krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260924 04:41:25.107862 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:25.238499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:25.353318 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:25.353603 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:25.353643 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:25.358568 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:25.358675 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:25.358704 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:25.358748 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:25.358783 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:25.364439 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:25.365592 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:25.366660 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:25.376178 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:25.376252 3427 server_base.cc:1034] running on GCE node
W20260924 04:41:25.376166 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:25.376180 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:25.376971 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:25.378172 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:25.379316 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790224885379287 us; error 43 us; skew 500 ppm
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:25.382177 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:25.383049 3427 webserver.cc:460] Webserver started at http://127.0.98.62:44347/ using document root <none> and password file <none>
I20260924 04:41:25.383644 3427 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:25.383704 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:25.383867 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:25.385911 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "9485230517b64f65b857a6c72ef89e7f"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "c2cafa3894a94380a4a05fdcdb6e313b"
server_key_iv: "3ba4fc9f58b2a8f8500d3af2dbb09b49"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.386431 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "9485230517b64f65b857a6c72ef89e7f"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "c2cafa3894a94380a4a05fdcdb6e313b"
server_key_iv: "3ba4fc9f58b2a8f8500d3af2dbb09b49"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.390249 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:41:25.392769 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.394129 3427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:25.394270 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "9485230517b64f65b857a6c72ef89e7f"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "c2cafa3894a94380a4a05fdcdb6e313b"
server_key_iv: "3ba4fc9f58b2a8f8500d3af2dbb09b49"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.394380 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:25.418499 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:25.422057 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:25.422267 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:25.429344 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39047
I20260924 04:41:25.429631 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39047 every 8 connection(s)
I20260924 04:41:25.430455 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:25.433355 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:25.435335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3427
I20260924 04:41:25.435468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:25.435736 392 external_mini_cluster.cc:1428] Setting key e8e0d012be8369aa8e8a75f6f1441b11
I20260924 04:41:25.440377 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f: Bootstrap starting.
I20260924 04:41:25.443264 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:25.444509 3495 log.cc:824] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:25.446975 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f: No bootstrap required, opened a new log
I20260924 04:41:25.451799 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:25.437121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.98.62:39047)
0924 04:41:25.437525 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:25.437543 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:25.437590 (+ 47us) server_negotiation.cc:407] Connection header received
0924 04:41:25.438233 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:25.438250 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:25.438589 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:25.438867 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:25.439959 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.441024 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.442333 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.442796 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.442945 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:25.446162 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:25.446168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:25.446185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:25.446216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:25.448259 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:25.448793 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:25.448799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:25.448805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:25.448953 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:25.449223 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:25.449228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:25.449231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:25.449842 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:25.450032 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:25.450251 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:41:25.450574 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260924 04:41:25.452078 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9485230517b64f65b857a6c72ef89e7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39047 } }
I20260924 04:41:25.452359 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:25.452510 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9485230517b64f65b857a6c72ef89e7f, State: Initialized, Role: FOLLOWER
I20260924 04:41:25.452988 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9485230517b64f65b857a6c72ef89e7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39047 } }
I20260924 04:41:25.453133 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:25.453199 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:25.453312 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:25.454356 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9485230517b64f65b857a6c72ef89e7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39047 } }
I20260924 04:41:25.454718 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9485230517b64f65b857a6c72ef89e7f; no voters:
I20260924 04:41:25.455032 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:25.455221 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:25.455482 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [term 1 LEADER]: Becoming Leader. State: Replica: 9485230517b64f65b857a6c72ef89e7f, State: Running, Role: LEADER
I20260924 04:41:25.456082 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9485230517b64f65b857a6c72ef89e7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39047 } }
I20260924 04:41:25.456314 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:25.457805 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9485230517b64f65b857a6c72ef89e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9485230517b64f65b857a6c72ef89e7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39047 } } }
I20260924 04:41:25.457974 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:25.457973 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9485230517b64f65b857a6c72ef89e7f. Latest consensus state: current_term: 1 leader_uuid: "9485230517b64f65b857a6c72ef89e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9485230517b64f65b857a6c72ef89e7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39047 } } }
I20260924 04:41:25.458321 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f [sys.catalog]: This master's current role is: LEADER
W20260924 04:41:25.461370 3515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:41:25.461536 3515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:41:25.461648 3514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:25.462424 3514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:25.469084 3514 catalog_manager.cc:1347] Generated new cluster ID: 456410b2415c493bba3e099514a67ae9
I20260924 04:41:25.469187 3514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:25.479795 3514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:25.480957 3514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:25.491562 3514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9485230517b64f65b857a6c72ef89e7f: Generated new TSK 0
I20260924 04:41:25.492341 3514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:25.531529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39047
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:25.647831 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:25.648159 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:25.648236 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:25.652390 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:25.652494 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:25.652588 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:25.658401 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:25.659605 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:25.660656 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:25.670719 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:25.670704 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:25.670755 3521 server_base.cc:1034] running on GCE node
W20260924 04:41:25.670704 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:25.671433 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:25.672086 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:25.673270 3521 hybrid_clock.cc:648] HybridClock initialized: now 1790224885673247 us; error 35 us; skew 500 ppm
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:25.676175 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:25.677066 3521 webserver.cc:460] Webserver started at http://127.0.98.1:36401/ using document root <none> and password file <none>
I20260924 04:41:25.677654 3521 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:25.677742 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:25.677965 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:25.680119 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "df2999508b8a4e57b305983fed0af0f0"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "0ea8de66ef366b3a8c3ff26b8198fba3"
server_key_iv: "6301aa09f1a19eef928e6ccf6e7319e5"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.680677 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "df2999508b8a4e57b305983fed0af0f0"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "0ea8de66ef366b3a8c3ff26b8198fba3"
server_key_iv: "6301aa09f1a19eef928e6ccf6e7319e5"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.684554 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:25.687054 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.688619 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:25.688862 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "df2999508b8a4e57b305983fed0af0f0"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "0ea8de66ef366b3a8c3ff26b8198fba3"
server_key_iv: "6301aa09f1a19eef928e6ccf6e7319e5"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.688979 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:25.704440 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:25.708007 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:25.708216 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:25.708827 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:25.709682 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:25.709750 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.709820 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:25.709859 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.719211 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43605
I20260924 04:41:25.719208 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43605 every 8 connection(s)
I20260924 04:41:25.720374 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:25.728796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3521
I20260924 04:41:25.728967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:25.729322 392 external_mini_cluster.cc:1428] Setting key 2482f44cc51c4110a615d841abb2d189
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:25.735963 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:25.722551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48159 (local address 127.0.98.62:39047)
0924 04:41:25.722696 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:25.722699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:25.723578 (+ 879us) server_negotiation.cc:407] Connection header received
0924 04:41:25.725010 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:25.725013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:25.725064 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:25.725147 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:25.726396 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.726963 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.728270 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.728514 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.728639 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:25.731612 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:25.731618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:25.731625 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:25.731661 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:25.733309 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:25.734034 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:25.734037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:25.734039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:25.734121 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:25.734559 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:25.734561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:25.734562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:25.734741 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:25.734817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:25.735669 (+ 852us) server_negotiation.cc:299] Negotiation successful
0924 04:41:25.735829 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:41:25.737680 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:25.722932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39047 (local address 127.0.98.1:48159)
0924 04:41:25.723409 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:25.723452 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:25.724777 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:25.725331 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:25.725338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:25.725721 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:25.726255 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:25.726269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.727097 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.727100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:25.728144 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:25.728155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.728717 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.728722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:25.728966 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:25.729725 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:25.729748 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:25.731429 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:25.733434 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:25.733444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:25.733468 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:25.733909 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:25.734214 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:25.734220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:25.734225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:25.734431 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:25.734848 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:25.734858 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:25.735288 (+ 430us) client_negotiation.cc:769] Sending connection context
0924 04:41:25.735563 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 04:41:25.736017 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":154,"threads_started":1}
I20260924 04:41:25.739378 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39047
I20260924 04:41:25.739665 3650 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:25.740260 3650 heartbeater.cc:511] Master 127.0.98.62:39047 requested a full tablet report, sending...
I20260924 04:41:25.741955 3459 ts_manager.cc:194] Registered new tserver with Master: df2999508b8a4e57b305983fed0af0f0 (127.0.98.1:43605)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:25.743310 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:25.758708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39047
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:25.872684 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:25.873034 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:25.873111 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:25.877354 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:25.877463 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:25.877564 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:25.883247 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:25.884555 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:25.885675 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:25.896112 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:25.896102 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:25.896207 3655 server_base.cc:1034] running on GCE node
W20260924 04:41:25.896103 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:25.896823 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:25.897769 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:25.899001 3655 hybrid_clock.cc:648] HybridClock initialized: now 1790224885898965 us; error 68 us; skew 500 ppm
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:25.902081 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:25.903113 3655 webserver.cc:460] Webserver started at http://127.0.98.2:36145/ using document root <none> and password file <none>
I20260924 04:41:25.903766 3655 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:25.903829 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:25.904055 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:25.906132 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "5959f096b1a349cb83a5b2f4b082cb5e"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "b3fe222bcae86de8632f360e5b332171"
server_key_iv: "6afef679e08b34e7149228effca3ab45"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.906713 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5959f096b1a349cb83a5b2f4b082cb5e"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "b3fe222bcae86de8632f360e5b332171"
server_key_iv: "6afef679e08b34e7149228effca3ab45"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.910808 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 04:41:25.913532 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.914888 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:41:25.915037 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "5959f096b1a349cb83a5b2f4b082cb5e"
format_stamp: "Formatted at 2026-09-24 04:41:25 on dist-test-slave-npg2"
server_key: "b3fe222bcae86de8632f360e5b332171"
server_key_iv: "6afef679e08b34e7149228effca3ab45"
server_key_version: "encryptionkey@0"
I20260924 04:41:25.915181 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:25.930286 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:25.933483 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:25.933703 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:25.934316 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:25.935230 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:25.935295 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.935361 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:25.935410 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:25.945020 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36731
I20260924 04:41:25.945084 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36731 every 8 connection(s)
I20260924 04:41:25.946054 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:25.955588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3655
I20260924 04:41:25.955719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:25.956064 392 external_mini_cluster.cc:1428] Setting key 99d40801e0c247c249051c2471190b5b
Sep 24 04:41:25 dist-test-slave-npg2 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:25.962997 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:25.948378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50913 (local address 127.0.98.62:39047)
0924 04:41:25.948599 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:25.948604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:25.949820 (+ 1216us) server_negotiation.cc:407] Connection header received
0924 04:41:25.951539 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:25.951543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:25.951594 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:25.951676 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:25.953084 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.953656 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.954980 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.955265 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.955355 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:25.958592 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:25.958599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:25.958605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:25.958647 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:25.960479 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:25.960997 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:25.961004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:25.961009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:25.961117 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:25.961518 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:25.961524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:25.961529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:25.961799 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:25.961922 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:25.962590 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 04:41:25.962741 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 04:41:25.963975 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:25.948941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39047 (local address 127.0.98.2:50913)
0924 04:41:25.949643 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:25.949691 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:25.951300 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:25.951858 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:25.951865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:25.952284 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:25.952915 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:25.952930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.953809 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.953812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:25.954835 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:25.954845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:25.955439 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:25.955444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:25.955690 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:25.956417 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:25.956441 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:25.958375 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:25.960586 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:25.960591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:25.960606 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:25.960874 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:25.961240 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:25.961242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:25.961244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:25.961401 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:25.962001 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:25.962011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:25.962307 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 04:41:25.962521 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 04:41:25.962833 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":128,"threads_started":1}
I20260924 04:41:25.965535 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39047
I20260924 04:41:25.965801 3784 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:41:25.966382 3784 heartbeater.cc:511] Master 127.0.98.62:39047 requested a full tablet report, sending...
I20260924 04:41:25.967744 3459 ts_manager.cc:194] Registered new tserver with Master: 5959f096b1a349cb83a5b2f4b082cb5e (127.0.98.2:36731)
I20260924 04:41:25.968428 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:25.986650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39047
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:26.101927 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:26.102213 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:26.102262 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:26.106379 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:26.106454 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:26.106515 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:26.112109 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:26.113268 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:26.114315 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:26.124639 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:26.124733 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:26.124810 3789 server_base.cc:1034] running on GCE node
W20260924 04:41:26.124639 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:26.125283 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:26.125974 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:26.127206 3789 hybrid_clock.cc:648] HybridClock initialized: now 1790224886127189 us; error 36 us; skew 500 ppm
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:26.130053 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:26.130972 3789 webserver.cc:460] Webserver started at http://127.0.98.3:39375/ using document root <none> and password file <none>
I20260924 04:41:26.131640 3789 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:26.131733 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:26.131968 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:26.134078 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "3fa36a10e33d4418874fa9dd11fe30ef"
format_stamp: "Formatted at 2026-09-24 04:41:26 on dist-test-slave-npg2"
server_key: "2215702146d7234d3e21137300e2cf86"
server_key_iv: "d31a8179c2e3435c5805dc71cdd9cfe1"
server_key_version: "encryptionkey@0"
I20260924 04:41:26.134644 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3fa36a10e33d4418874fa9dd11fe30ef"
format_stamp: "Formatted at 2026-09-24 04:41:26 on dist-test-slave-npg2"
server_key: "2215702146d7234d3e21137300e2cf86"
server_key_iv: "d31a8179c2e3435c5805dc71cdd9cfe1"
server_key_version: "encryptionkey@0"
I20260924 04:41:26.138567 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:41:26.141253 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:26.142814 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:26.142977 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "3fa36a10e33d4418874fa9dd11fe30ef"
format_stamp: "Formatted at 2026-09-24 04:41:26 on dist-test-slave-npg2"
server_key: "2215702146d7234d3e21137300e2cf86"
server_key_iv: "d31a8179c2e3435c5805dc71cdd9cfe1"
server_key_version: "encryptionkey@0"
I20260924 04:41:26.143106 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:26.159921 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:26.163326 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:26.163533 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:26.164151 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:26.164994 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:26.165060 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:26.165129 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:26.165176 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:26.174364 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44247
I20260924 04:41:26.174381 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44247 every 8 connection(s)
I20260924 04:41:26.175479 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:26.183868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3789
I20260924 04:41:26.184069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:26.184345 392 external_mini_cluster.cc:1428] Setting key 083f5a0b6cfd0967140b39592ac8e5ac
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:26.190366 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:26.177549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53021 (local address 127.0.98.62:39047)
0924 04:41:26.177693 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:26.177696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:26.178595 (+ 899us) server_negotiation.cc:407] Connection header received
0924 04:41:26.180199 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:26.180202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:26.180261 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:26.180366 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:26.181633 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:26.182202 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:26.183442 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:26.183679 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:26.183791 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:26.186778 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:26.186782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:26.186784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:26.186815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:26.188441 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:26.188946 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:26.188949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:26.188950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:26.189026 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:26.189325 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:26.189327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:26.189328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:26.189504 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:26.189581 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:26.190131 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 04:41:26.190241 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:41:26.191340 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:26.177974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39047 (local address 127.0.98.3:53021)
0924 04:41:26.178434 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:26.178480 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:26.179940 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:26.180541 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:26.180550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:26.180930 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:26.181473 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:26.181497 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:26.182349 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:26.182351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:26.183318 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:26.183327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:26.183836 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:26.183840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:26.184044 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:26.184722 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:26.184744 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:26.186570 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:26.188550 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:26.188555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:26.188568 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:26.188838 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:26.189104 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:26.189106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:26.189107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:26.189250 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:26.189614 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:26.189620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:26.189887 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 04:41:26.190069 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:41:26.190345 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
I20260924 04:41:26.192556 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39047
I20260924 04:41:26.192898 3918 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:26.193367 3918 heartbeater.cc:511] Master 127.0.98.62:39047 requested a full tablet report, sending...
I20260924 04:41:26.194422 3459 ts_manager.cc:194] Registered new tserver with Master: 3fa36a10e33d4418874fa9dd11fe30ef (127.0.98.3:44247)
I20260924 04:41:26.195022 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53021
I20260924 04:41:26.198415 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224886, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224886, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:26.224254 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:26.215477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.0.98.62:39047)
0924 04:41:26.215617 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:26.215621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:26.215783 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:41:26.215903 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:26.215905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:26.215962 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:26.216047 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:26.216774 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:26.217313 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:26.218218 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:26.218452 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:26.218555 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:26.221311 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:26.221314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:26.221317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:26.221342 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:26.222827 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:26.223237 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:26.223239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:26.223240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:26.223314 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:26.223532 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:26.223534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:26.223535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:26.223753 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:26.223855 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:26.224038 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:41:26.224133 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:41:26.224640 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56844
I20260924 04:41:26.225719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3521
I20260924 04:41:26.231438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3655
I20260924 04:41:26.236765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3789
I20260924 04:41:26.242496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3427
2026-09-24T04:41:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2243 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3930](info): set up 2 sockets
Sep 24 04:41:26 dist-test-slave-npg2 krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260924 04:41:27.332664 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:27 dist-test-slave-npg2 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:27.564819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:27.685431 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:27.685750 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:27.685792 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:27.690122 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:27.690210 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:27.690230 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:27.690248 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:27.690264 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:27.696170 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:27.697371 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:27.698488 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:27.707412 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:27.707506 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:27.707510 3943 server_base.cc:1034] running on GCE node
W20260924 04:41:27.707424 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:27.708318 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:27.709580 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:27.710748 3943 hybrid_clock.cc:648] HybridClock initialized: now 1790224887710699 us; error 65 us; skew 500 ppm
Sep 24 04:41:27 dist-test-slave-npg2 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:27.714136 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:27.715273 3943 webserver.cc:460] Webserver started at http://127.0.98.62:34473/ using document root <none> and password file <none>
I20260924 04:41:27.715893 3943 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:27.715961 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:27.716224 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:27.718590 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "b10be650d748480595a46ed08c052f5b"
format_stamp: "Formatted at 2026-09-24 04:41:27 on dist-test-slave-npg2"
server_key: "86eb58685f9fac52233402edf6dc14bb"
server_key_iv: "d064f87d50c03e9c863d2903a6d53824"
server_key_version: "encryptionkey@0"
I20260924 04:41:27.719313 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "b10be650d748480595a46ed08c052f5b"
format_stamp: "Formatted at 2026-09-24 04:41:27 on dist-test-slave-npg2"
server_key: "86eb58685f9fac52233402edf6dc14bb"
server_key_iv: "d064f87d50c03e9c863d2903a6d53824"
server_key_version: "encryptionkey@0"
I20260924 04:41:27.723884 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:41:27.726482 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:27.727864 3943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:27.728000 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "b10be650d748480595a46ed08c052f5b"
format_stamp: "Formatted at 2026-09-24 04:41:27 on dist-test-slave-npg2"
server_key: "86eb58685f9fac52233402edf6dc14bb"
server_key_iv: "d064f87d50c03e9c863d2903a6d53824"
server_key_version: "encryptionkey@0"
I20260924 04:41:27.728132 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:27.742450 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:27.746030 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:27.746248 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:27.753697 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39613
I20260924 04:41:27.753719 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39613 every 8 connection(s)
I20260924 04:41:27.754705 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:27.757536 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:27.761896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 3943
I20260924 04:41:27.762001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:27.762298 392 external_mini_cluster.cc:1428] Setting key acc1724275b58678091e28c7dcf63e91
I20260924 04:41:27.763621 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b: Bootstrap starting.
I20260924 04:41:27.766530 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:27.767496 4011 log.cc:824] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:27.769737 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b: No bootstrap required, opened a new log
Sep 24 04:41:27 dist-test-slave-npg2 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:27.774544 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10be650d748480595a46ed08c052f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39613 } }
I20260924 04:41:27.774847 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:27.774909 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10be650d748480595a46ed08c052f5b, State: Initialized, Role: FOLLOWER
I20260924 04:41:27.775471 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10be650d748480595a46ed08c052f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39613 } }
I20260924 04:41:27.775637 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:27.775712 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:27.775836 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:27.777092 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10be650d748480595a46ed08c052f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39613 } }
I20260924 04:41:27.777474 4011 leader_election.cc:302] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b10be650d748480595a46ed08c052f5b; no voters:
I20260924 04:41:27.777814 4011 leader_election.cc:288] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:27.777940 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:27.778192 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [term 1 LEADER]: Becoming Leader. State: Replica: b10be650d748480595a46ed08c052f5b, State: Running, Role: LEADER
I20260924 04:41:27.778581 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:27.763862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.0.98.62:39613)
0924 04:41:27.764286 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:27.764305 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:27.764347 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:41:27.765285 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:27.765309 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:27.765821 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:27.766201 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:27.767013 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:27.768445 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:27.769368 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:27.770071 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:27.770260 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:27.773132 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:27.773140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:27.773160 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:27.773196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:27.775215 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:27.775675 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:27.775683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:27.775690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:27.775867 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:27.776189 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:27.776192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:27.776194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:27.776749 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:27.776955 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:27.777175 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:41:27.777494 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260924 04:41:27.778533 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10be650d748480595a46ed08c052f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39613 } }
I20260924 04:41:27.779182 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:27.780726 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b10be650d748480595a46ed08c052f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10be650d748480595a46ed08c052f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39613 } } }
I20260924 04:41:27.780755 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b10be650d748480595a46ed08c052f5b. Latest consensus state: current_term: 1 leader_uuid: "b10be650d748480595a46ed08c052f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10be650d748480595a46ed08c052f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39613 } } }
I20260924 04:41:27.780848 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:27.780859 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:27.781386 4027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:27.784744 4027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:27.790202 4027 catalog_manager.cc:1347] Generated new cluster ID: ef922906fd064ee48bde1ea5fc4dbb04
I20260924 04:41:27.790287 4027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:27.811955 4027 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:27.813419 4027 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:27.824843 4027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b10be650d748480595a46ed08c052f5b: Generated new TSK 0
I20260924 04:41:27.825795 4027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:27.866807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39613
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:27.984465 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:27.984781 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:27.984862 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:27.989140 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:27.989239 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:27.989334 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:27.995510 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:27.996750 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:27.997758 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:28.007884 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:28.007990 4037 server_base.cc:1034] running on GCE node
W20260924 04:41:28.008015 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:28.007884 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:28.008737 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:28.009728 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:28.010907 4037 hybrid_clock.cc:648] HybridClock initialized: now 1790224888010876 us; error 44 us; skew 500 ppm
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:28.014008 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:28.014956 4037 webserver.cc:460] Webserver started at http://127.0.98.1:40173/ using document root <none> and password file <none>
I20260924 04:41:28.015597 4037 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:28.015688 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:28.015918 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:28.018527 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "d339aa947e6d4bbe94540289894d7a33"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "32752bd085cc5e416d7a6baac05ef713"
server_key_iv: "9fe3734667e05d3bf4d75b8db502484d"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.019212 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d339aa947e6d4bbe94540289894d7a33"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "32752bd085cc5e416d7a6baac05ef713"
server_key_iv: "9fe3734667e05d3bf4d75b8db502484d"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.023419 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:28.026187 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.027612 4037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:28.027752 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "d339aa947e6d4bbe94540289894d7a33"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "32752bd085cc5e416d7a6baac05ef713"
server_key_iv: "9fe3734667e05d3bf4d75b8db502484d"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.027870 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:28.046761 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:28.050279 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:28.050555 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:28.051352 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:28.052249 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:28.052318 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.052390 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:28.052429 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.061861 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45741
I20260924 04:41:28.061887 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45741 every 8 connection(s)
I20260924 04:41:28.062830 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:28.064007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4037
I20260924 04:41:28.064133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:28.064431 392 external_mini_cluster.cc:1428] Setting key 185f01faafe6746b47504180ea74dd39
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:28.078842 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.065127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52215 (local address 127.0.98.62:39613)
0924 04:41:28.065260 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:28.065265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:28.066295 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 04:41:28.067733 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:28.067737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:28.067791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:28.067871 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:28.069114 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.069653 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.070831 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.071084 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.071242 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.074209 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:28.074212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:28.074214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:28.074240 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:28.076082 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.076708 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.076711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.076712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.076815 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.077351 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.077355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.077356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.077740 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:28.077856 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:28.078419 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 04:41:28.078595 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:41:28.080053 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.065572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39613 (local address 127.0.98.1:52215)
0924 04:41:28.066114 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:28.066170 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:28.067522 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:28.068050 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:28.068064 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:28.068452 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:28.068956 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:28.068971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.069803 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.069806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:28.070699 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:28.070708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.071293 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.071297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:28.071476 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.072199 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:28.072216 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:28.074007 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:28.076227 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:28.076232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:28.076246 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:28.076552 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:28.076920 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:28.076925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:28.076928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:28.077153 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:28.077865 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:28.077871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:28.078164 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 04:41:28.078332 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:41:28.078705 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":168,"threads_started":1}
I20260924 04:41:28.081794 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39613
I20260924 04:41:28.082095 4166 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:28.082806 4166 heartbeater.cc:511] Master 127.0.98.62:39613 requested a full tablet report, sending...
I20260924 04:41:28.084630 3975 ts_manager.cc:194] Registered new tserver with Master: d339aa947e6d4bbe94540289894d7a33 (127.0.98.1:45741)
I20260924 04:41:28.085812 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:28.094216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39613
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:28.207548 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:28.207823 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:28.207866 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:28.212044 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:28.212113 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:28.212193 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:28.218497 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:28.219784 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:28.220853 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:28.231106 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:28.231293 4171 server_base.cc:1034] running on GCE node
W20260924 04:41:28.231148 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:28.231252 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:28.231907 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:28.232712 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:28.233913 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790224888233873 us; error 49 us; skew 500 ppm
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:28.237005 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:28.237995 4171 webserver.cc:460] Webserver started at http://127.0.98.2:38877/ using document root <none> and password file <none>
I20260924 04:41:28.238607 4171 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:28.238698 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:28.238929 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:28.241156 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff4ba75bc8294baba828e62191e36478"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "3202222858505f07069b1908409364cf"
server_key_iv: "906db035a71a104c469b98604d011198"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.241711 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff4ba75bc8294baba828e62191e36478"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "3202222858505f07069b1908409364cf"
server_key_iv: "906db035a71a104c469b98604d011198"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.246035 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:41:28.248594 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.249989 4171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:28.250137 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "ff4ba75bc8294baba828e62191e36478"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "3202222858505f07069b1908409364cf"
server_key_iv: "906db035a71a104c469b98604d011198"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.250283 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:28.266141 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:28.269438 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:28.269667 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:28.270293 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:28.271265 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:28.271332 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.271409 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:28.271448 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.280973 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42063
I20260924 04:41:28.280997 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42063 every 8 connection(s)
I20260924 04:41:28.282028 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:28.291033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4171
I20260924 04:41:28.291198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:28.291474 392 external_mini_cluster.cc:1428] Setting key 18280802727a752d2cb133226ab94ee5
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:28.296515 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.284074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45601 (local address 127.0.98.62:39613)
0924 04:41:28.284212 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:28.284215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:28.285092 (+ 877us) server_negotiation.cc:407] Connection header received
0924 04:41:28.286508 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:28.286511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:28.286567 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:28.286668 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:28.288058 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.288598 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.289751 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.289984 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.290103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.292852 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:28.292856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:28.292858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:28.292882 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:28.294377 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.294916 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.294919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.294921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.295005 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.295387 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.295392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.295394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.295587 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:28.295695 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:28.296207 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 04:41:28.296343 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:28.297356 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.284465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39613 (local address 127.0.98.2:45601)
0924 04:41:28.284918 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:28.284981 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:28.286288 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:28.286843 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:28.286850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:28.287292 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:28.287902 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:28.287916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.288727 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.288735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:28.289624 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:28.289633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.290147 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.290151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:28.290321 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.290988 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:28.291005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:28.292689 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:28.294517 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:28.294524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:28.294541 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:28.294810 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:28.295106 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:28.295109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:28.295111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:28.295292 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:28.295691 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:28.295697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:28.295961 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 04:41:28.296154 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:41:28.296487 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260924 04:41:28.298702 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39613
I20260924 04:41:28.298947 4300 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:28.299500 4300 heartbeater.cc:511] Master 127.0.98.62:39613 requested a full tablet report, sending...
I20260924 04:41:28.300612 3975 ts_manager.cc:194] Registered new tserver with Master: ff4ba75bc8294baba828e62191e36478 (127.0.98.2:42063)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:41:28.301226 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:28.322144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39613
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:28.436455 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:28.436722 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:28.436766 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:28.440950 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:28.441008 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:28.441093 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:28.446348 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:28.447556 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:28.448621 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:28.458380 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:28.458429 4305 server_base.cc:1034] running on GCE node
W20260924 04:41:28.458369 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:28.458370 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:28.459048 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:28.459738 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:28.460911 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790224888460872 us; error 48 us; skew 500 ppm
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:28.463907 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:28.464859 4305 webserver.cc:460] Webserver started at http://127.0.98.3:33845/ using document root <none> and password file <none>
I20260924 04:41:28.465449 4305 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:28.465515 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:28.465791 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:28.467926 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e778a2a7bcb448ba0e5a2cbc512267b"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "460e8f07d9d75f4c22d87b30f7a2177a"
server_key_iv: "f7984e71f0d31b646bf72413aabe98fc"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.468482 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e778a2a7bcb448ba0e5a2cbc512267b"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "460e8f07d9d75f4c22d87b30f7a2177a"
server_key_iv: "f7984e71f0d31b646bf72413aabe98fc"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.472255 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:28.474722 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.476045 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:28.476143 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "3e778a2a7bcb448ba0e5a2cbc512267b"
format_stamp: "Formatted at 2026-09-24 04:41:28 on dist-test-slave-npg2"
server_key: "460e8f07d9d75f4c22d87b30f7a2177a"
server_key_iv: "f7984e71f0d31b646bf72413aabe98fc"
server_key_version: "encryptionkey@0"
I20260924 04:41:28.476275 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:28.496148 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:28.499910 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:28.500180 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:28.500782 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:28.501621 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:28.501684 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.501760 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:28.501803 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:28.510371 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44849
I20260924 04:41:28.510428 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44849 every 8 connection(s)
I20260924 04:41:28.511371 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:28.519332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4305
I20260924 04:41:28.519562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:28.519874 392 external_mini_cluster.cc:1428] Setting key 6c24a52df3fd756608f2511add883d50
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:28.525928 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.513278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32839 (local address 127.0.98.62:39613)
0924 04:41:28.513420 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:28.513424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:28.514300 (+ 876us) server_negotiation.cc:407] Connection header received
0924 04:41:28.515831 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:28.515834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:28.515894 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:28.516000 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:28.517278 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.517820 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.519003 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.519337 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.519456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.522256 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:28.522259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:28.522261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:28.522285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:28.523770 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.524243 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.524246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.524247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.524318 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.524768 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.524771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.524772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.524945 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:28.525025 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:28.525558 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 04:41:28.525718 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:41:28.526810 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.513685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39613 (local address 127.0.98.3:32839)
0924 04:41:28.514147 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:28.514191 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:28.515604 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:28.516203 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:28.516214 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:28.516602 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:28.517124 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:28.517139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.517950 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.517953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:28.518870 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:28.518879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.519511 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.519515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:28.519739 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.520411 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:28.520435 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:28.522092 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:28.523877 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:28.523883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:28.523896 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:28.524159 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:28.524536 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:28.524538 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:28.524540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:28.524683 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:28.525056 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:28.525061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:28.525322 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:41:28.525505 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 04:41:28.525874 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260924 04:41:28.528172 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39613
I20260924 04:41:28.528393 4434 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:28.528834 4434 heartbeater.cc:511] Master 127.0.98.62:39613 requested a full tablet report, sending...
I20260924 04:41:28.529841 3975 ts_manager.cc:194] Registered new tserver with Master: 3e778a2a7bcb448ba0e5a2cbc512267b (127.0.98.3:44849)
I20260924 04:41:28.530370 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32839
I20260924 04:41:28.533917 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:28.560463 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.550068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.0.98.1:45741)
0924 04:41:28.550336 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:28.550350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:28.550379 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:41:28.550429 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:28.550434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:28.550683 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:28.550807 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:28.551560 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.552363 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.553193 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.553631 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.553716 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.556602 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:28.556613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:28.556628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:28.556657 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:28.558480 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.558928 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.558934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.558935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.559018 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.559366 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.559370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.559371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.559637 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:28.559816 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:28.560103 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 04:41:28.560263 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:28.571982 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:28.563643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.62:39613)
0924 04:41:28.563768 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:28.563772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:28.563787 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:41:28.563880 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:28.563883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:28.563948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:28.564012 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:28.564632 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.565212 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.566005 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:28.566238 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:28.566324 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:28.568964 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:28.568967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:28.568969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:28.568994 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:28.570333 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.570749 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.570752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.570753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.570823 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:28.571047 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:28.571049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:28.571050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:28.571357 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:28.571462 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:28.571674 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 04:41:28.571772 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 04:41:28.572395 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52854
I20260924 04:41:28.573027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4037
I20260924 04:41:28.578861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4171
I20260924 04:41:28.584486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4305
I20260924 04:41:28.590173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 3943
2026-09-24T04:41:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2347 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[4446](info): set up 2 sockets
Sep 24 04:41:28 dist-test-slave-npg2 krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260924 04:41:29.679422 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:29 dist-test-slave-npg2 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:29.804368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41943
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:29.916360 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:29.916680 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:29.916754 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:29.920914 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:29.920995 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:29.921032 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:29.921092 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:29.921128 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:29.926357 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:29.927500 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:29.928524 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:29.937151 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:29.937168 4459 server_base.cc:1034] running on GCE node
W20260924 04:41:29.937208 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:29.937208 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:29.937994 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:29.939061 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:29.940279 4459 hybrid_clock.cc:648] HybridClock initialized: now 1790224889940249 us; error 43 us; skew 500 ppm
Sep 24 04:41:29 dist-test-slave-npg2 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:29.949051 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:29.949887 4459 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:29.949975 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:29.950223 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:29.952481 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "bbbab3ab63a248b7868b72aa8f674aba"
format_stamp: "Formatted at 2026-09-24 04:41:29 on dist-test-slave-npg2"
server_key: "763fefb6ce3e910f09abcc61f2b426a8"
server_key_iv: "39ffba2c6a5a9170f7d55c33545bdefd"
server_key_version: "encryptionkey@0"
I20260924 04:41:29.953048 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbbab3ab63a248b7868b72aa8f674aba"
format_stamp: "Formatted at 2026-09-24 04:41:29 on dist-test-slave-npg2"
server_key: "763fefb6ce3e910f09abcc61f2b426a8"
server_key_iv: "39ffba2c6a5a9170f7d55c33545bdefd"
server_key_version: "encryptionkey@0"
I20260924 04:41:29.956789 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:41:29.959295 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:29.960525 4459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:29.960661 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "bbbab3ab63a248b7868b72aa8f674aba"
format_stamp: "Formatted at 2026-09-24 04:41:29 on dist-test-slave-npg2"
server_key: "763fefb6ce3e910f09abcc61f2b426a8"
server_key_iv: "39ffba2c6a5a9170f7d55c33545bdefd"
server_key_version: "encryptionkey@0"
I20260924 04:41:29.960768 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:30.016258 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:30.019888 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:30.020124 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:30.027838 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41943
I20260924 04:41:30.028116 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41943 every 8 connection(s)
I20260924 04:41:30.028586 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:30.031486 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:30.031702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4459
I20260924 04:41:30.031831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:30.032158 392 external_mini_cluster.cc:1428] Setting key 5c15c59ce414bb252381e64bd89e0c82
I20260924 04:41:30.037942 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: Bootstrap starting.
I20260924 04:41:30.040318 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: Neither blocks nor log segments found. Creating new log.
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:30.041085 4526 log.cc:824] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:30.043087 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: No bootstrap required, opened a new log
I20260924 04:41:30.045920 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.033552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.0.98.62:41943)
0924 04:41:30.034017 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.034033 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.034069 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:30.034738 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.034756 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.035095 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.035558 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:30.036321 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.037402 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.038327 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.038814 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.038978 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.041526 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:30.041531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:30.041544 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:30.041570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:30.043276 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.043681 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.043687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.043693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.043806 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.044084 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.044087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.044089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.044468 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:30.044685 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.044911 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.045201 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":137,"threads_started":1}
I20260924 04:41:30.046759 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbab3ab63a248b7868b72aa8f674aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41943 } }
I20260924 04:41:30.047039 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:30.047091 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbab3ab63a248b7868b72aa8f674aba, State: Initialized, Role: FOLLOWER
I20260924 04:41:30.047580 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbab3ab63a248b7868b72aa8f674aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41943 } }
I20260924 04:41:30.047740 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:30.047796 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:30.047890 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:30.049050 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbab3ab63a248b7868b72aa8f674aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41943 } }
I20260924 04:41:30.049350 4526 leader_election.cc:302] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbab3ab63a248b7868b72aa8f674aba; no voters:
I20260924 04:41:30.049621 4526 leader_election.cc:288] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:30.049738 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:30.049966 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [term 1 LEADER]: Becoming Leader. State: Replica: bbbab3ab63a248b7868b72aa8f674aba, State: Running, Role: LEADER
I20260924 04:41:30.050276 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbab3ab63a248b7868b72aa8f674aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41943 } }
I20260924 04:41:30.050951 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:30.051967 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbbab3ab63a248b7868b72aa8f674aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbab3ab63a248b7868b72aa8f674aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41943 } } }
I20260924 04:41:30.052102 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:30.052382 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbab3ab63a248b7868b72aa8f674aba. Latest consensus state: current_term: 1 leader_uuid: "bbbab3ab63a248b7868b72aa8f674aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbab3ab63a248b7868b72aa8f674aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41943 } } }
I20260924 04:41:30.052496 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba [sys.catalog]: This master's current role is: LEADER
W20260924 04:41:30.055929 4546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:41:30.056005 4546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:41:30.056075 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:30.056746 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:30.062762 4540 catalog_manager.cc:1347] Generated new cluster ID: 46b86c9a16d44053ba558310915a921b
I20260924 04:41:30.062850 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:30.070515 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:30.071931 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:30.077322 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: Generated new TSK 0
I20260924 04:41:30.078162 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:30.108270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41943
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:30.232761 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:30.233012 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:30.233058 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:30.237176 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:30.237238 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:30.237298 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:30.242412 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:30.243520 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:30.244580 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:30.254024 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:30.254148 4552 server_base.cc:1034] running on GCE node
W20260924 04:41:30.254066 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:30.254040 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:30.254691 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:30.255416 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:30.256605 4552 hybrid_clock.cc:648] HybridClock initialized: now 1790224890256585 us; error 37 us; skew 500 ppm
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:30.259594 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:30.260324 4552 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:30.260412 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:30.260637 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:30.262709 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "223194ffd4ac4c63a4a183868a86ccdd"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "7fc73a7dac7377d852d5cd061267ca34"
server_key_iv: "ec6bcf8ccc014d5539b1e66128a734fd"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.263312 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "223194ffd4ac4c63a4a183868a86ccdd"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "7fc73a7dac7377d852d5cd061267ca34"
server_key_iv: "ec6bcf8ccc014d5539b1e66128a734fd"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.267271 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:41:30.269726 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.271003 4552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:30.271222 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "223194ffd4ac4c63a4a183868a86ccdd"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "7fc73a7dac7377d852d5cd061267ca34"
server_key_iv: "ec6bcf8ccc014d5539b1e66128a734fd"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.271342 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:30.286382 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:30.289443 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:30.289601 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:30.290119 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:30.290935 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:30.290982 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.291023 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:30.291038 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.299847 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35709
I20260924 04:41:30.299865 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35709 every 8 connection(s)
I20260924 04:41:30.300465 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:30.304898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4552
I20260924 04:41:30.305044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:30.305330 392 external_mini_cluster.cc:1428] Setting key 55ed105786595df278ffe72c384de01e
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:30.315068 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.302346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41995 (local address 127.0.98.62:41943)
0924 04:41:30.302507 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.302511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.303631 (+ 1120us) server_negotiation.cc:407] Connection header received
0924 04:41:30.305016 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.305020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.305079 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.305163 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:30.306485 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.307018 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.308239 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.308474 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.308605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.311431 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:30.311434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:30.311436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:30.311461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:30.312989 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.313522 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.313526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.313528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.313612 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.313960 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.313963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.313965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.314154 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:30.314256 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.314767 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.314900 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:41:30.315979 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.302758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41943 (local address 127.0.98.1:41995)
0924 04:41:30.303419 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.303465 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.304790 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.305390 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.305419 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.305816 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:30.306334 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.306349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.307179 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.307187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.308110 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.308119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.308663 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.308666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.308833 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.309464 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:30.309480 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:30.311257 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:30.313128 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:30.313134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:30.313151 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:30.313416 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:30.313712 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:30.313715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:30.313717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:30.313862 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:30.314254 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:30.314261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:30.314525 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.314713 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.315062 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":112,"threads_started":1}
I20260924 04:41:30.317274 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:30.317569 4680 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:30.318142 4680 heartbeater.cc:511] Master 127.0.98.62:41943 requested a full tablet report, sending...
I20260924 04:41:30.319988 4490 ts_manager.cc:194] Registered new tserver with Master: 223194ffd4ac4c63a4a183868a86ccdd (127.0.98.1:35709)
I20260924 04:41:30.321273 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:30.334224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41943
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:30.447587 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:30.447850 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:30.447896 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:30.452229 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:30.452299 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:30.452415 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:30.457901 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:30.459049 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:30.460206 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:30.469818 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:30.469861 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:30.469956 4685 server_base.cc:1034] running on GCE node
W20260924 04:41:30.469818 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:30.470592 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:30.471349 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:30.472522 4685 hybrid_clock.cc:648] HybridClock initialized: now 1790224890472484 us; error 55 us; skew 500 ppm
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:30.475699 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:30.476457 4685 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:30.476521 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:30.476743 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:30.478830 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "6384743c7172430ba8e76bc4d825a2d1"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "69c7d759290a67e9adf014485a9e9db5"
server_key_iv: "78877b2df55d557a349ae853f15615df"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.479463 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6384743c7172430ba8e76bc4d825a2d1"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "69c7d759290a67e9adf014485a9e9db5"
server_key_iv: "78877b2df55d557a349ae853f15615df"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.483273 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:41:30.485827 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.487044 4685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:30.487212 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "6384743c7172430ba8e76bc4d825a2d1"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "69c7d759290a67e9adf014485a9e9db5"
server_key_iv: "78877b2df55d557a349ae853f15615df"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.487319 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:30.506009 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:30.509480 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:30.509692 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:30.510347 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:30.511265 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:30.511312 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.511354 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:30.511453 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.520547 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38203
I20260924 04:41:30.520553 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38203 every 8 connection(s)
I20260924 04:41:30.521169 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:30.530841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4685
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:30.530980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:30.531329 392 external_mini_cluster.cc:1428] Setting key 43edfd7303204dc387da3e6270b4b79f
I20260924 04:41:30.535566 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.523031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44325 (local address 127.0.98.62:41943)
0924 04:41:30.523274 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.523278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.524080 (+ 802us) server_negotiation.cc:407] Connection header received
0924 04:41:30.525509 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.525513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.525569 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.525683 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:30.526868 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.527462 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.528604 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.528840 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.528952 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.531660 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:30.531664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:30.531667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:30.531700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:30.533324 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.533912 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.533919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.533923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.534016 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.534354 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.534359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.534363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.534615 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:30.534694 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.535352 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.535449 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:41:30.536842 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.523473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41943 (local address 127.0.98.2:44325)
0924 04:41:30.523929 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.523973 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.525289 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.525856 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.525864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.526231 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:30.526724 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.526737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.527585 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.527594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.528480 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.528489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.528992 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.528997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.529174 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.529782 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:30.529799 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:30.531447 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:30.533447 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:30.533451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:30.533501 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:30.533764 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:30.534114 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:30.534116 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:30.534125 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:30.534265 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:30.534717 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:30.534724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:30.535064 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.535324 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.535717 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260924 04:41:30.538069 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41943
I20260924 04:41:30.538311 4813 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:30.538806 4813 heartbeater.cc:511] Master 127.0.98.62:41943 requested a full tablet report, sending...
I20260924 04:41:30.539924 4490 ts_manager.cc:194] Registered new tserver with Master: 6384743c7172430ba8e76bc4d825a2d1 (127.0.98.2:38203)
I20260924 04:41:30.540493 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44325
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:30.560660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41943
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:30.689416 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:30.689762 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:30.689849 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:30.694643 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:30.694757 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:30.694864 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:30.700506 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:30.701619 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:30.702728 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:30.712350 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:30.712359 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:30.712446 4818 server_base.cc:1034] running on GCE node
W20260924 04:41:30.712368 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:30.713088 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:30.713665 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:30.714815 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790224890714785 us; error 46 us; skew 500 ppm
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:30.717471 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:30.718140 4818 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:30.718200 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:30.718355 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:30.720405 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "a70e6394b3a34d2ca02cb77e599dc21c"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "a4343059bcbd00dd4c0ab368a00f5e5c"
server_key_iv: "b1845beac248de934738c6cd8efebd93"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.720896 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a70e6394b3a34d2ca02cb77e599dc21c"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "a4343059bcbd00dd4c0ab368a00f5e5c"
server_key_iv: "b1845beac248de934738c6cd8efebd93"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.724354 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:41:30.726660 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.727882 4818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:30.728005 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "a70e6394b3a34d2ca02cb77e599dc21c"
format_stamp: "Formatted at 2026-09-24 04:41:30 on dist-test-slave-npg2"
server_key: "a4343059bcbd00dd4c0ab368a00f5e5c"
server_key_iv: "b1845beac248de934738c6cd8efebd93"
server_key_version: "encryptionkey@0"
I20260924 04:41:30.728109 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:30.742342 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:30.745519 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:30.745719 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:30.746277 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:30.747073 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:30.747169 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.747249 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:30.747294 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:30.756821 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41897
I20260924 04:41:30.756843 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41897 every 8 connection(s)
I20260924 04:41:30.757416 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:30.762202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4818
I20260924 04:41:30.762333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:30.762594 392 external_mini_cluster.cc:1428] Setting key 8e1e1a7396972af7662099428a257476
Sep 24 04:41:30 dist-test-slave-npg2 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:30.771359 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.759263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52381 (local address 127.0.98.62:41943)
0924 04:41:30.759413 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.759417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.760274 (+ 857us) server_negotiation.cc:407] Connection header received
0924 04:41:30.761612 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.761614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.761659 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.761734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:30.763031 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.763813 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.765171 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.765397 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.765582 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.768075 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:30.768078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:30.768080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:30.768119 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:30.769353 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.769929 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.769932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.769933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.770002 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.770301 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.770303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.770304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.770464 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:30.770537 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.771091 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.771242 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:41:30.772315 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.759664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41943 (local address 127.0.98.3:52381)
0924 04:41:30.760124 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.760167 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.761401 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.761902 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.761908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.762291 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:30.762869 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.762883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.763958 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.763962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.765033 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.765043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.765614 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.765618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.765785 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.766346 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:30.766363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:30.767905 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:30.769475 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:30.769480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:30.769492 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:30.769840 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:30.770083 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:30.770085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:30.770086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:30.770223 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:30.770625 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:30.770631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:30.770863 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.771052 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.771422 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":103,"threads_started":1}
I20260924 04:41:30.773392 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41943
I20260924 04:41:30.773725 4946 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:30.774200 4946 heartbeater.cc:511] Master 127.0.98.62:41943 requested a full tablet report, sending...
I20260924 04:41:30.775343 4490 ts_manager.cc:194] Registered new tserver with Master: a70e6394b3a34d2ca02cb77e599dc21c (127.0.98.3:41897)
I20260924 04:41:30.776201 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52381
I20260924 04:41:30.776434 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:41:30.786208 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.778821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.98.62:41943)
0924 04:41:30.778969 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.778974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.779158 (+ 184us) server_negotiation.cc:407] Connection header received
0924 04:41:30.779309 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.779311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.779349 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.779407 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:30.780189 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.780689 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.781537 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.781768 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.781850 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.783230 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:30.783233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:30.783235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:30.783257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:30.784494 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.784934 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.784937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.784939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.785032 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:30.785304 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:30.785307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:30.785308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:30.785540 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:30.785686 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.785907 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.786008 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:41:30.789247 4490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:30.792536 4490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:30.807261 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.801000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35709 (local address 127.0.0.1:58314)
0924 04:41:30.801422 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.801477 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.801605 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.802367 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.802372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.802400 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:30.802562 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.802573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.804273 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.804276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.805757 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.805768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.806881 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.806886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.806968 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.806987 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.807056 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.807136 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260924 04:41:30.807396 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.801213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.0.98.1:35709)
0924 04:41:30.801815 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.801821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.801845 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:41:30.801927 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.801932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.802169 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.802733 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:30.802882 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.804121 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.805884 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.806705 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.806852 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.806960 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.807100 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.807227 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":95,"threads_started":1}
I20260924 04:41:30.807467 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.801971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.0.98.3:41897)
0924 04:41:30.802392 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.802403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.802698 (+ 295us) server_negotiation.cc:407] Connection header received
0924 04:41:30.802812 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.802817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.803006 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.803175 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:30.803408 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.804422 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.805920 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.806492 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.806613 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.806760 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.807211 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.807311 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
I20260924 04:41:30.807646 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.801819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.0.98.2:38203)
0924 04:41:30.802235 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.802247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.802447 (+ 200us) server_negotiation.cc:407] Connection header received
0924 04:41:30.802941 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.802945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.803105 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.803283 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:30.803631 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.804423 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.805967 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.806705 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.806825 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.806960 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.807326 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.807426 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":79,"threads_started":1}
I20260924 04:41:30.807672 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.801808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41897 (local address 127.0.0.1:41016)
0924 04:41:30.802575 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.802585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.802701 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.803166 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.803169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.803186 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:30.803296 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.803302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.804554 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.804559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.805766 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.805775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.806628 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.806631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.806737 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.806774 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.806906 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.807028 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":85,"thread_start_us":80,"threads_started":1}
I20260924 04:41:30.808063 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.801020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.0.1:45330)
0924 04:41:30.802361 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.802374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.802813 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.803426 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.803429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.803443 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:30.803511 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.803514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.804566 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.804569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.805818 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.805828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.807045 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.807051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.807161 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.807191 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.807858 (+ 667us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.807929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260924 04:41:30.810554 4880 tablet_service.cc:1467] Processing CreateTablet for tablet f6fbf98c522c4079b43825d3ecd3b667 (DEFAULT_TABLE table=test-table [id=68cbff6655b24b08a5f1fd6a6eb4840c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:30.810729 4747 tablet_service.cc:1467] Processing CreateTablet for tablet f6fbf98c522c4079b43825d3ecd3b667 (DEFAULT_TABLE table=test-table [id=68cbff6655b24b08a5f1fd6a6eb4840c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:30.810911 4614 tablet_service.cc:1467] Processing CreateTablet for tablet f6fbf98c522c4079b43825d3ecd3b667 (DEFAULT_TABLE table=test-table [id=68cbff6655b24b08a5f1fd6a6eb4840c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:30.811677 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6fbf98c522c4079b43825d3ecd3b667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:30.811764 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6fbf98c522c4079b43825d3ecd3b667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:30.811764 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6fbf98c522c4079b43825d3ecd3b667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:30.817806 4963 tablet_bootstrap.cc:492] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: Bootstrap starting.
I20260924 04:41:30.819020 4964 tablet_bootstrap.cc:492] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: Bootstrap starting.
I20260924 04:41:30.819396 4965 tablet_bootstrap.cc:492] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Bootstrap starting.
I20260924 04:41:30.820889 4963 tablet_bootstrap.cc:654] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:30.820919 4964 tablet_bootstrap.cc:654] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:30.821424 4965 tablet_bootstrap.cc:654] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:30.821816 4964 log.cc:824] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:30.822081 4963 log.cc:824] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:30.822580 4965 log.cc:824] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:30.823860 4964 tablet_bootstrap.cc:492] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: No bootstrap required, opened a new log
I20260924 04:41:30.824064 4964 ts_tablet_manager.cc:1397] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:41:30.824121 4963 tablet_bootstrap.cc:492] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: No bootstrap required, opened a new log
I20260924 04:41:30.824357 4963 ts_tablet_manager.cc:1397] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:41:30.824416 4965 tablet_bootstrap.cc:492] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: No bootstrap required, opened a new log
I20260924 04:41:30.824616 4965 ts_tablet_manager.cc:1397] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:41:30.827397 4964 raft_consensus.cc:348] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.827649 4964 raft_consensus.cc:374] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:30.827723 4964 raft_consensus.cc:729] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223194ffd4ac4c63a4a183868a86ccdd, State: Initialized, Role: FOLLOWER
I20260924 04:41:30.828195 4964 consensus_queue.cc:260] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.828316 4965 raft_consensus.cc:348] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.828661 4965 raft_consensus.cc:374] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:30.828843 4965 raft_consensus.cc:729] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6384743c7172430ba8e76bc4d825a2d1, State: Initialized, Role: FOLLOWER
I20260924 04:41:30.828867 4680 heartbeater.cc:503] Master 127.0.98.62:41943 was elected leader, sending a full tablet report...
I20260924 04:41:30.829268 4964 ts_tablet_manager.cc:1428] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:41:30.829365 4965 consensus_queue.cc:260] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.829291 4963 raft_consensus.cc:348] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.829568 4963 raft_consensus.cc:374] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:30.829633 4963 raft_consensus.cc:729] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70e6394b3a34d2ca02cb77e599dc21c, State: Initialized, Role: FOLLOWER
I20260924 04:41:30.830144 4813 heartbeater.cc:503] Master 127.0.98.62:41943 was elected leader, sending a full tablet report...
I20260924 04:41:30.830094 4963 consensus_queue.cc:260] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.830459 4965 ts_tablet_manager.cc:1428] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260924 04:41:30.830969 4963 ts_tablet_manager.cc:1428] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:41:30.831333 4946 heartbeater.cc:503] Master 127.0.98.62:41943 was elected leader, sending a full tablet report...
I20260924 04:41:30.992689 4971 raft_consensus.cc:482] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:30.992937 4971 raft_consensus.cc:504] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:30.994149 4971 leader_election.cc:288] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223194ffd4ac4c63a4a183868a86ccdd (127.0.98.1:35709), 6384743c7172430ba8e76bc4d825a2d1 (127.0.98.2:38203)
I20260924 04:41:30.998554 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.994420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35709 (local address 127.0.98.3:49767)
0924 04:41:30.994740 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.994756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.994866 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.995229 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.995234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.995259 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:30.995395 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.995401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.996371 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.996374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.997247 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.997256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.998079 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.998083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.998195 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.998222 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.998306 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.998393 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260924 04:41:30.998586 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.994515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49767 (local address 127.0.98.1:35709)
0924 04:41:30.994659 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.994664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.994839 (+ 175us) server_negotiation.cc:407] Connection header received
0924 04:41:30.995017 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.995021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.995094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.995198 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:30.995587 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.996238 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.997390 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.997929 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.998012 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.998099 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.998363 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.998457 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:41:30.998953 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.994583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46187 (local address 127.0.98.2:38203)
0924 04:41:30.994709 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:30.994714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:30.995201 (+ 487us) server_negotiation.cc:407] Connection header received
0924 04:41:30.995261 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:30.995264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:30.995312 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:30.995401 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:30.995843 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.996359 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.997755 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.998293 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.998391 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.998461 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:30.998762 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 04:41:30.998842 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:41:30.999326 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:30.994578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.98.3:46187)
0924 04:41:30.994986 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:30.995037 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:30.995174 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:30.995585 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:30.995588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:30.995606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:30.995714 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.995720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.996692 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.996695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.997619 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:30.997629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:30.998417 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:30.998420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:30.998529 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:30.998625 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 04:41:30.999105 (+ 480us) client_negotiation.cc:240] Negotiation successful
0924 04:41:30.999217 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":135,"threads_started":1}
I20260924 04:41:30.999387 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6fbf98c522c4079b43825d3ecd3b667" candidate_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223194ffd4ac4c63a4a183868a86ccdd" is_pre_election: true
I20260924 04:41:30.999761 4634 raft_consensus.cc:2454] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70e6394b3a34d2ca02cb77e599dc21c in term 0.
I20260924 04:41:30.999959 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6fbf98c522c4079b43825d3ecd3b667" candidate_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6384743c7172430ba8e76bc4d825a2d1" is_pre_election: true
I20260924 04:41:31.000303 4767 raft_consensus.cc:2454] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70e6394b3a34d2ca02cb77e599dc21c in term 0.
I20260924 04:41:31.000322 4833 leader_election.cc:302] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223194ffd4ac4c63a4a183868a86ccdd, a70e6394b3a34d2ca02cb77e599dc21c; no voters:
I20260924 04:41:31.000630 4971 raft_consensus.cc:2781] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:31.000735 4971 raft_consensus.cc:482] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:31.000799 4971 raft_consensus.cc:3037] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:31.001847 4971 raft_consensus.cc:504] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
I20260924 04:41:31.002264 4971 leader_election.cc:288] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [CANDIDATE]: Term 1 election: Requested vote from peers 223194ffd4ac4c63a4a183868a86ccdd (127.0.98.1:35709), 6384743c7172430ba8e76bc4d825a2d1 (127.0.98.2:38203)
I20260924 04:41:31.002642 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6fbf98c522c4079b43825d3ecd3b667" candidate_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223194ffd4ac4c63a4a183868a86ccdd"
I20260924 04:41:31.002781 4634 raft_consensus.cc:3037] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:31.002912 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6fbf98c522c4079b43825d3ecd3b667" candidate_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6384743c7172430ba8e76bc4d825a2d1"
I20260924 04:41:31.003067 4767 raft_consensus.cc:3037] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:31.003964 4634 raft_consensus.cc:2454] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70e6394b3a34d2ca02cb77e599dc21c in term 1.
I20260924 04:41:31.004213 4767 raft_consensus.cc:2454] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70e6394b3a34d2ca02cb77e599dc21c in term 1.
I20260924 04:41:31.004387 4833 leader_election.cc:302] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223194ffd4ac4c63a4a183868a86ccdd, a70e6394b3a34d2ca02cb77e599dc21c; no voters:
I20260924 04:41:31.004592 4971 raft_consensus.cc:2781] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:31.004832 4971 raft_consensus.cc:686] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 1 LEADER]: Becoming Leader. State: Replica: a70e6394b3a34d2ca02cb77e599dc21c, State: Running, Role: LEADER
I20260924 04:41:31.005275 4971 consensus_queue.cc:237] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } }
W20260924 04:41:31.008426 4947 tablet.cc:2369] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:31.009073 4489 catalog_manager.cc:5495] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c reported cstate change: term changed from 0 to 1, leader changed from <none> to a70e6394b3a34d2ca02cb77e599dc21c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a70e6394b3a34d2ca02cb77e599dc21c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } health_report { overall_health: HEALTHY } } }
W20260924 04:41:31.022246 4814 tablet.cc:2369] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:31.046711 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.042957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.3:41897)
0924 04:41:31.043068 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.043072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.043086 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:31.043285 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.043287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.043345 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.043434 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:31.043708 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:31.044210 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:31.045145 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:31.045348 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:31.045421 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:31.045794 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:31.046292 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:31.046387 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.046461 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.046512 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 04:41:31.051949 4681 tablet.cc:2369] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:31.052651 4634 raft_consensus.cc:1261] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd [term 1 FOLLOWER]: Refusing update from remote peer a70e6394b3a34d2ca02cb77e599dc21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:31.052691 4767 raft_consensus.cc:1261] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 1 FOLLOWER]: Refusing update from remote peer a70e6394b3a34d2ca02cb77e599dc21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:31.053436 4971 consensus_queue.cc:1035] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6384743c7172430ba8e76bc4d825a2d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:31.053752 4973 consensus_queue.cc:1035] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "223194ffd4ac4c63a4a183868a86ccdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:31.062711 4977 mvcc.cc:205] Tried to move back new op lower bound from 7332761153743634432 to 7332761153560883200. Current Snapshot: MvccSnapshot[applied={T|T < 7332761153743634432}]
I20260924 04:41:31.069389 4490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:31.069571 4490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:31.072016 4490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbbab3ab63a248b7868b72aa8f674aba: Sending DeleteTablet for 3 replicas of tablet f6fbf98c522c4079b43825d3ecd3b667
I20260924 04:41:31.072892 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet f6fbf98c522c4079b43825d3ecd3b667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58314
I20260924 04:41:31.072930 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet f6fbf98c522c4079b43825d3ecd3b667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45330
I20260924 04:41:31.073400 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet f6fbf98c522c4079b43825d3ecd3b667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41016
I20260924 04:41:31.073462 4987 tablet_replica.cc:330] T f6fbf98c522c4079b43825d3ecd3b667 P 223194ffd4ac4c63a4a183868a86ccdd: stopping tablet replica
I20260924 04:41:31.073498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4552
I20260924 04:41:31.073709 4986 tablet_replica.cc:330] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: stopping tablet replica
I20260924 04:41:31.074211 4986 raft_consensus.cc:2229] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:31.074532 4986 raft_consensus.cc:2258] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:31.076116 4986 ts_tablet_manager.cc:1905] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:31.079068 4986 ts_tablet_manager.cc:1918] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:31.079236 4986 log.cc:1196] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/f6fbf98c522c4079b43825d3ecd3b667
I20260924 04:41:31.079599 4986 ts_tablet_manager.cc:1939] T f6fbf98c522c4079b43825d3ecd3b667 P 6384743c7172430ba8e76bc4d825a2d1: Deleting consensus metadata
W20260924 04:41:31.080395 4474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35709 (error 108)
I20260924 04:41:31.080476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4685
I20260924 04:41:31.082055 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.081784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35709 (local address 127.0.0.1:58322)
0924 04:41:31.081900 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.081974 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:41:31.082297 4474 catalog_manager.cc:4584] TS 223194ffd4ac4c63a4a183868a86ccdd (127.0.98.1:35709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6fbf98c522c4079b43825d3ecd3b667: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35709: connect: Connection refused (error 111)
I20260924 04:41:31.086934 4988 tablet_replica.cc:330] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c: stopping tablet replica
I20260924 04:41:31.086994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4818
I20260924 04:41:31.087391 4988 raft_consensus.cc:2229] T f6fbf98c522c4079b43825d3ecd3b667 P a70e6394b3a34d2ca02cb77e599dc21c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:31.088873 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.088596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.0.1:45338)
0924 04:41:31.088755 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.088806 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 04:41:31.092896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4459
2026-09-24T04:41:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2503 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:41:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:31Z Disabled control of system clock
I20260924 04:41:31.123513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40419
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:41:31.235363 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:31.235638 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:31.235677 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:31.239878 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:31.239960 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:31.239976 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:31.239993 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:31.240010 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:31.245031 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:31.246142 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:31.247543 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:41:31.256639 4993 server_base.cc:1034] running on GCE node
W20260924 04:41:31.256595 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.256601 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.256620 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:31.257467 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:31.258728 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:31.259893 4993 hybrid_clock.cc:648] HybridClock initialized: now 1790224891259858 us; error 49 us; skew 500 ppm
I20260924 04:41:31.261978 4993 webserver.cc:460] Webserver started at http://127.0.98.62:45729/ using document root <none> and password file <none>
I20260924 04:41:31.262707 4993 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:31.262776 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:31.263010 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:31.265177 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "4a1a31ee49ef41e1abb445bc004fd67a"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "e2bd61bfb6c5178f16cd4811693ce057"
server_key_iv: "3d77e2d3a625249867e2d19f458b80fa"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.265755 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a1a31ee49ef41e1abb445bc004fd67a"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "e2bd61bfb6c5178f16cd4811693ce057"
server_key_iv: "3d77e2d3a625249867e2d19f458b80fa"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.270098 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:41:31.272858 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.274567 4993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:31.274745 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "4a1a31ee49ef41e1abb445bc004fd67a"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "e2bd61bfb6c5178f16cd4811693ce057"
server_key_iv: "3d77e2d3a625249867e2d19f458b80fa"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.274876 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:31.279763 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:31.280397 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:31.280583 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:31.287508 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40419
I20260924 04:41:31.287518 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40419 every 8 connection(s)
I20260924 04:41:31.288471 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:31.289140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 4993
I20260924 04:41:31.289279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:31.289541 392 external_mini_cluster.cc:1428] Setting key c8974b959cef3da53ce7623b4316ca7d
I20260924 04:41:31.291929 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:31.296010 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.291082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.98.62:40419)
0924 04:41:31.292363 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.292378 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.292415 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:41:31.293138 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.293157 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.293521 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.293885 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.294725 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.294730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.294742 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.294752 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.294805 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.294822 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.294978 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.295049 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.295321 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":690,"threads_started":1}
I20260924 04:41:31.297930 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a: Bootstrap starting.
I20260924 04:41:31.300226 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:31.300964 5060 log.cc:824] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:31.303018 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a: No bootstrap required, opened a new log
I20260924 04:41:31.306741 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260924 04:41:31.306991 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:31.307080 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1a31ee49ef41e1abb445bc004fd67a, State: Initialized, Role: FOLLOWER
I20260924 04:41:31.307545 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260924 04:41:31.307723 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:31.307811 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:31.307987 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:31.308981 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260924 04:41:31.309368 5060 leader_election.cc:302] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a1a31ee49ef41e1abb445bc004fd67a; no voters:
I20260924 04:41:31.309718 5060 leader_election.cc:288] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:31.309815 5065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:31.310066 5065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [term 1 LEADER]: Becoming Leader. State: Replica: 4a1a31ee49ef41e1abb445bc004fd67a, State: Running, Role: LEADER
I20260924 04:41:31.310369 5065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260924 04:41:31.310956 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:31.312148 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } } }
I20260924 04:41:31.312155 5067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a1a31ee49ef41e1abb445bc004fd67a. Latest consensus state: current_term: 1 leader_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a31ee49ef41e1abb445bc004fd67a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } } }
I20260924 04:41:31.312295 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:31.312299 5067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:31.312651 5074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:31.315768 5074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:31.321704 5074 catalog_manager.cc:1347] Generated new cluster ID: dbc6b60e064742e18269309700f95ec7
I20260924 04:41:31.321801 5074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:31.348445 5074 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:31.348836 5074 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:31.356244 5074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a: Generated new TSK 0
I20260924 04:41:31.357087 5074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:41:31.361963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40419
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:41:31.484267 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:31.484525 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:31.484566 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:31.488816 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:31.488878 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:31.488972 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:31.493765 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:31.494868 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:31.496312 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:31.507565 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.507565 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.507622 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:31.507867 5084 server_base.cc:1034] running on GCE node
I20260924 04:41:31.508255 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:31.508881 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:31.510044 5084 hybrid_clock.cc:648] HybridClock initialized: now 1790224891510021 us; error 37 us; skew 500 ppm
I20260924 04:41:31.512079 5084 webserver.cc:460] Webserver started at http://127.0.98.1:38825/ using document root <none> and password file <none>
I20260924 04:41:31.512729 5084 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:31.512795 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:31.513011 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:31.515066 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c45ba20016d4dc8b04dd99d35bf76c1"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "8ea558bc18b16a8689510b4289acb5d1"
server_key_iv: "51d879906e01c0359e41c02b6ffed153"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.515688 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c45ba20016d4dc8b04dd99d35bf76c1"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "8ea558bc18b16a8689510b4289acb5d1"
server_key_iv: "51d879906e01c0359e41c02b6ffed153"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.519498 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:41:31.521957 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.523268 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:31.523391 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "1c45ba20016d4dc8b04dd99d35bf76c1"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "8ea558bc18b16a8689510b4289acb5d1"
server_key_iv: "51d879906e01c0359e41c02b6ffed153"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.523521 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:31.528103 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:31.528728 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:31.528867 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:31.529357 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:31.530196 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:31.530241 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.530277 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:31.530293 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.539377 5084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46151
I20260924 04:41:31.539404 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46151 every 8 connection(s)
I20260924 04:41:31.540318 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:31.546305 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.542086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36159 (local address 127.0.98.62:40419)
0924 04:41:31.542253 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.542258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.543411 (+ 1153us) server_negotiation.cc:407] Connection header received
0924 04:41:31.544749 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.544752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.544788 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.544864 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.545901 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.545904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.545906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.545917 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.545937 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.545945 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.546021 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.546165 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.546202 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:41:31.547217 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.542511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40419 (local address 127.0.98.1:36159)
0924 04:41:31.543234 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.543275 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.544550 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:31.545018 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:31.545024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:31.545137 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:31.545659 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:31.545660 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:31.545699 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:31.545764 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:31.546020 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:31.546025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:31.546029 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:41:31.546184 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:41:31.546366 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260924 04:41:31.548385 5212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40419
I20260924 04:41:31.548741 5212 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:31.548878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5084
I20260924 04:41:31.548997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:31.549259 5212 heartbeater.cc:511] Master 127.0.98.62:40419 requested a full tablet report, sending...
I20260924 04:41:31.549302 392 external_mini_cluster.cc:1428] Setting key a48f7296329b40aca37b2168a3869ffb
I20260924 04:41:31.550817 5023 ts_manager.cc:194] Registered new tserver with Master: 1c45ba20016d4dc8b04dd99d35bf76c1 (127.0.98.1:46151)
I20260924 04:41:31.551347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40419
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:41:31.663976 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:31.664243 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:31.664283 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:31.668524 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:31.668581 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:31.668646 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:31.673458 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:31.674527 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:31.675969 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:31.685667 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.685761 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.685807 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:31.685943 5215 server_base.cc:1034] running on GCE node
I20260924 04:41:31.686417 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:31.687106 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:31.688320 5215 hybrid_clock.cc:648] HybridClock initialized: now 1790224891688305 us; error 28 us; skew 500 ppm
I20260924 04:41:31.690337 5215 webserver.cc:460] Webserver started at http://127.0.98.2:42307/ using document root <none> and password file <none>
I20260924 04:41:31.690930 5215 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:31.690991 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:31.691293 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:31.693361 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a1e5ff97c28496abda071390bc53979"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "1d2d9c83aa81d8699a23467ece22c774"
server_key_iv: "14c0bf8697592cc7475c0ea61c6bdc36"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.693926 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a1e5ff97c28496abda071390bc53979"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "1d2d9c83aa81d8699a23467ece22c774"
server_key_iv: "14c0bf8697592cc7475c0ea61c6bdc36"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.697661 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260924 04:41:31.700261 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.701769 5215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:31.701987 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "1a1e5ff97c28496abda071390bc53979"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "1d2d9c83aa81d8699a23467ece22c774"
server_key_iv: "14c0bf8697592cc7475c0ea61c6bdc36"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.702131 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:31.706877 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:31.707543 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:31.707726 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:31.708267 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:31.709163 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:31.709233 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.709306 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:31.709348 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.718603 5215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46861
I20260924 04:41:31.718626 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46861 every 8 connection(s)
I20260924 04:41:31.719590 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:31.725678 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.721428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52933 (local address 127.0.98.62:40419)
0924 04:41:31.721623 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.721627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.722782 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 04:41:31.724047 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.724050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.724086 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.724161 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.725280 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.725283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.725284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.725295 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.725314 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.725322 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.725397 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.725520 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.725557 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 04:41:31.726737 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.721916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40419 (local address 127.0.98.2:52933)
0924 04:41:31.722636 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.722678 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.723856 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:31.724359 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:31.724365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:31.724452 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:31.725030 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:31.725031 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:31.725071 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:31.725139 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:31.725395 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:31.725400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:31.725404 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:41:31.725547 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 04:41:31.725732 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260924 04:41:31.727653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5215
I20260924 04:41:31.727799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:31.728065 5343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40419
I20260924 04:41:31.728139 392 external_mini_cluster.cc:1428] Setting key 3707b6a980abf243b0096c54e408ed5e
I20260924 04:41:31.728309 5343 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:31.728746 5343 heartbeater.cc:511] Master 127.0.98.62:40419 requested a full tablet report, sending...
I20260924 04:41:31.729862 5023 ts_manager.cc:194] Registered new tserver with Master: 1a1e5ff97c28496abda071390bc53979 (127.0.98.2:46861)
I20260924 04:41:31.730532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40419
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:41:31.842422 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:31.842710 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:31.842751 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:31.846698 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:31.846757 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:31.846815 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:31.851555 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:31.852614 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:31.853955 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:31.863559 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:31.863662 5346 server_base.cc:1034] running on GCE node
W20260924 04:41:31.863546 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:31.863552 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:31.864274 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:31.865010 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:31.866179 5346 hybrid_clock.cc:648] HybridClock initialized: now 1790224891866144 us; error 45 us; skew 500 ppm
I20260924 04:41:31.868155 5346 webserver.cc:460] Webserver started at http://127.0.98.3:33821/ using document root <none> and password file <none>
I20260924 04:41:31.868834 5346 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:31.868922 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:31.869215 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:31.871255 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "61e5925e3f3c4b04b2e3d151ddb0b742"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "bb7e71d67f3744852b082b086bc0acb0"
server_key_iv: "b879b02e76822930bf0ebd445daa9ab4"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.871793 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61e5925e3f3c4b04b2e3d151ddb0b742"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "bb7e71d67f3744852b082b086bc0acb0"
server_key_iv: "b879b02e76822930bf0ebd445daa9ab4"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.875406 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:41:31.877846 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.879051 5346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:31.879251 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "61e5925e3f3c4b04b2e3d151ddb0b742"
format_stamp: "Formatted at 2026-09-24 04:41:31 on dist-test-slave-npg2"
server_key: "bb7e71d67f3744852b082b086bc0acb0"
server_key_iv: "b879b02e76822930bf0ebd445daa9ab4"
server_key_version: "encryptionkey@0"
I20260924 04:41:31.879370 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:31.883836 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:31.884414 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:31.884584 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:31.885098 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:31.885979 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:31.886045 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.886107 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:31.886153 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:31.895315 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43003
I20260924 04:41:31.895334 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43003 every 8 connection(s)
I20260924 04:41:31.896260 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:31.896507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5346
I20260924 04:41:31.896651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:31.896960 392 external_mini_cluster.cc:1428] Setting key 91545bfc551d6eaf0122012241ea869a
I20260924 04:41:31.902453 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.898154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46109 (local address 127.0.98.62:40419)
0924 04:41:31.898284 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.898288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.899643 (+ 1355us) server_negotiation.cc:407] Connection header received
0924 04:41:31.900907 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.900911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.900945 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.901019 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.902056 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.902058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.902060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.902071 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.902091 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.902098 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.902184 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.902311 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.902348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:41:31.903409 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.898663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40419 (local address 127.0.98.3:46109)
0924 04:41:31.899454 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.899520 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.900706 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:31.901192 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:31.901199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:31.901288 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:31.901814 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:31.901816 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:31.901856 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:31.901923 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:31.902189 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:31.902195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:31.902199 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:41:31.902335 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 04:41:31.902552 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260924 04:41:31.904650 5474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40419
I20260924 04:41:31.904899 5474 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:31.905355 5474 heartbeater.cc:511] Master 127.0.98.62:40419 requested a full tablet report, sending...
I20260924 04:41:31.906435 5023 ts_manager.cc:194] Registered new tserver with Master: 61e5925e3f3c4b04b2e3d151ddb0b742 (127.0.98.3:43003)
I20260924 04:41:31.910694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:41:31.914467 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.912747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.0.98.62:40419)
0924 04:41:31.912901 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.912906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.913065 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:41:31.913217 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.913220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.913262 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.913345 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.914076 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.914079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.914082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.914093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.914112 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.914135 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.914223 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.914297 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.914340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:41:31.917145 5023 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:31.920171 5023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:31.931526 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.929396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.98.1:46151)
0924 04:41:31.929728 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.929736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.929943 (+ 207us) server_negotiation.cc:407] Connection header received
0924 04:41:31.930021 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.930026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.930227 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.930381 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.930997 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.931004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.931019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.931032 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.931065 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.931089 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.931232 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.931316 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.931384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":144,"threads_started":1}
I20260924 04:41:31.931635 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.929004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.0.1:38698)
0924 04:41:31.929846 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.929859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.929937 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:31.930361 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:31.930364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:31.930378 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:31.930786 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:31.930788 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:31.930798 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:31.930823 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:31.931212 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:31.931216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:31.931219 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:41:31.931303 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:41:31.931337 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":93,"threads_started":1}
I20260924 04:41:31.931941 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.928274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46861 (local address 127.0.0.1:34136)
0924 04:41:31.928759 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.928803 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.928898 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:31.929870 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:31.929875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:31.929966 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:31.930598 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:31.930599 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:31.930619 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:31.930702 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:31.931054 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:31.931066 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:31.931075 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:41:31.931211 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 04:41:31.931279 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":95,"threads_started":1}
I20260924 04:41:31.932150 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.928944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.0.98.2:46861)
0924 04:41:31.929288 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.929304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.929328 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:41:31.929496 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.929506 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.929718 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.929873 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.930849 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.930856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.930868 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.930880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.930924 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.930956 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.931067 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.931414 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.931493 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":139,"threads_started":1}
I20260924 04:41:31.934258 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.928411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43003 (local address 127.0.0.1:35590)
0924 04:41:31.929347 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.929361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.929420 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:31.931195 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:31.931198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:31.931212 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:31.932986 (+ 1774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:31.932988 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:31.932999 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:31.933024 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:31.934015 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:31.934019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:31.934022 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:41:31.934108 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:41:31.934151 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":110,"threads_started":1}
I20260924 04:41:31.934643 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.928568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.0.98.3:43003)
0924 04:41:31.930455 (+ 1887us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.930474 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.930499 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:41:31.930817 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.930823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.930997 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:31.933158 (+ 2161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:31.933723 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:31.933728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:31.933738 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:31.933747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:31.933780 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:31.933802 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:31.933908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:31.934414 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 04:41:31.934482 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1501,"thread_start_us":127,"threads_started":1}
I20260924 04:41:31.934640 5146 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0d37a6fb7342ffb070345bbd812faf (DEFAULT_TABLE table=test-table [id=9f79ed7c9cd64248a63c24f17ab7f7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:31.934600 5277 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0d37a6fb7342ffb070345bbd812faf (DEFAULT_TABLE table=test-table [id=9f79ed7c9cd64248a63c24f17ab7f7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:31.935753 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0d37a6fb7342ffb070345bbd812faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:31.935748 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0d37a6fb7342ffb070345bbd812faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:31.941540 5491 tablet_bootstrap.cc:492] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Bootstrap starting.
I20260924 04:41:31.941919 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0d37a6fb7342ffb070345bbd812faf (DEFAULT_TABLE table=test-table [id=9f79ed7c9cd64248a63c24f17ab7f7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:31.942710 5492 tablet_bootstrap.cc:492] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: Bootstrap starting.
I20260924 04:41:31.943009 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0d37a6fb7342ffb070345bbd812faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:31.944005 5491 tablet_bootstrap.cc:654] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:31.944775 5492 tablet_bootstrap.cc:654] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:31.944815 5491 log.cc:824] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:31.945585 5492 log.cc:824] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:31.946790 5491 tablet_bootstrap.cc:492] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: No bootstrap required, opened a new log
I20260924 04:41:31.947009 5491 ts_tablet_manager.cc:1397] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:41:31.947678 5492 tablet_bootstrap.cc:492] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: No bootstrap required, opened a new log
I20260924 04:41:31.947870 5492 ts_tablet_manager.cc:1397] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:41:31.950932 5491 raft_consensus.cc:348] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.951184 5495 tablet_bootstrap.cc:492] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Bootstrap starting.
I20260924 04:41:31.951237 5491 raft_consensus.cc:374] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:31.951292 5491 raft_consensus.cc:729] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1e5ff97c28496abda071390bc53979, State: Initialized, Role: FOLLOWER
I20260924 04:41:31.951752 5491 consensus_queue.cc:260] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.952387 5343 heartbeater.cc:503] Master 127.0.98.62:40419 was elected leader, sending a full tablet report...
I20260924 04:41:31.952723 5491 ts_tablet_manager.cc:1428] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:41:31.952858 5492 raft_consensus.cc:348] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.953140 5492 raft_consensus.cc:374] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:31.953209 5492 raft_consensus.cc:729] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c45ba20016d4dc8b04dd99d35bf76c1, State: Initialized, Role: FOLLOWER
I20260924 04:41:31.953450 5495 tablet_bootstrap.cc:654] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:31.953625 5492 consensus_queue.cc:260] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.954280 5212 heartbeater.cc:503] Master 127.0.98.62:40419 was elected leader, sending a full tablet report...
I20260924 04:41:31.954321 5495 log.cc:824] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:31.954586 5492 ts_tablet_manager.cc:1428] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:41:31.956634 5495 tablet_bootstrap.cc:492] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: No bootstrap required, opened a new log
I20260924 04:41:31.956820 5495 ts_tablet_manager.cc:1397] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:41:31.960361 5495 raft_consensus.cc:348] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.960559 5495 raft_consensus.cc:374] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:31.960592 5495 raft_consensus.cc:729] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e5925e3f3c4b04b2e3d151ddb0b742, State: Initialized, Role: FOLLOWER
I20260924 04:41:31.960911 5495 consensus_queue.cc:260] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.961458 5474 heartbeater.cc:503] Master 127.0.98.62:40419 was elected leader, sending a full tablet report...
I20260924 04:41:31.961751 5495 ts_tablet_manager.cc:1428] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 04:41:31.970496 5344 tablet.cc:2369] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:31.997560 5499 raft_consensus.cc:482] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:31.997907 5499 raft_consensus.cc:504] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:31.999316 5499 leader_election.cc:288] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1e5ff97c28496abda071390bc53979 (127.0.98.2:46861), 1c45ba20016d4dc8b04dd99d35bf76c1 (127.0.98.1:46151)
I20260924 04:41:32.001210 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.999644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54987 (local address 127.0.98.2:46861)
0924 04:41:31.999762 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.999769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:31.999786 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:41:31.999939 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:31.999944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:31.999996 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.000086 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:32.000795 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:32.000798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:32.000801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:32.000818 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:32.000836 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:32.000844 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:32.000918 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:32.001014 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:41:32.001057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:41:32.001215 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.999515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46861 (local address 127.0.98.3:54987)
0924 04:41:31.999716 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.999732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:31.999822 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:32.000115 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:32.000117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:32.000131 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:32.000659 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:32.000661 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:32.000670 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:32.000693 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:32.000925 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:32.000930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:32.000935 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:41:32.001019 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:41:32.001069 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 04:41:32.001436 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.999603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.98.3:39397)
0924 04:41:31.999918 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:31.999932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:32.000023 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:32.000273 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:32.000275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:32.000311 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:32.000844 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:41:32.000845 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:32.000852 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:41:32.000869 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:32.001162 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:32.001165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:32.001167 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:41:32.001248 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:41:32.001280 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
I20260924 04:41:32.001434 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:31.999725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39397 (local address 127.0.98.1:46151)
0924 04:41:31.999908 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:31.999913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.000019 (+ 106us) server_negotiation.cc:407] Connection header received
0924 04:41:32.000113 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.000116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.000154 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.000247 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:32.001015 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:32.001020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:32.001023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:32.001036 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:32.001055 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:32.001066 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:32.001167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:32.001247 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:41:32.001290 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 04:41:32.001982 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0d37a6fb7342ffb070345bbd812faf" candidate_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" is_pre_election: true
I20260924 04:41:32.001981 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0d37a6fb7342ffb070345bbd812faf" candidate_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1e5ff97c28496abda071390bc53979" is_pre_election: true
I20260924 04:41:32.002302 5166 raft_consensus.cc:2454] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e5925e3f3c4b04b2e3d151ddb0b742 in term 0.
I20260924 04:41:32.002311 5297 raft_consensus.cc:2454] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e5925e3f3c4b04b2e3d151ddb0b742 in term 0.
I20260924 04:41:32.002760 5363 leader_election.cc:302] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c45ba20016d4dc8b04dd99d35bf76c1, 61e5925e3f3c4b04b2e3d151ddb0b742; no voters:
I20260924 04:41:32.003023 5499 raft_consensus.cc:2781] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:32.003161 5499 raft_consensus.cc:482] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:32.003226 5499 raft_consensus.cc:3037] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:32.004658 5499 raft_consensus.cc:504] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:32.005020 5499 leader_election.cc:288] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [CANDIDATE]: Term 1 election: Requested vote from peers 1a1e5ff97c28496abda071390bc53979 (127.0.98.2:46861), 1c45ba20016d4dc8b04dd99d35bf76c1 (127.0.98.1:46151)
I20260924 04:41:32.005316 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0d37a6fb7342ffb070345bbd812faf" candidate_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1e5ff97c28496abda071390bc53979"
I20260924 04:41:32.005446 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0d37a6fb7342ffb070345bbd812faf" candidate_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1"
I20260924 04:41:32.005473 5297 raft_consensus.cc:3037] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:32.005587 5166 raft_consensus.cc:3037] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:32.006843 5166 raft_consensus.cc:2454] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e5925e3f3c4b04b2e3d151ddb0b742 in term 1.
I20260924 04:41:32.006843 5297 raft_consensus.cc:2454] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e5925e3f3c4b04b2e3d151ddb0b742 in term 1.
I20260924 04:41:32.007166 5363 leader_election.cc:302] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a1e5ff97c28496abda071390bc53979, 1c45ba20016d4dc8b04dd99d35bf76c1, 61e5925e3f3c4b04b2e3d151ddb0b742; no voters:
I20260924 04:41:32.007318 5499 raft_consensus.cc:2781] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:32.007548 5499 raft_consensus.cc:686] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 1 LEADER]: Becoming Leader. State: Replica: 61e5925e3f3c4b04b2e3d151ddb0b742, State: Running, Role: LEADER
I20260924 04:41:32.008141 5499 consensus_queue.cc:237] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 04:41:32.011790 5023 catalog_manager.cc:5495] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61e5925e3f3c4b04b2e3d151ddb0b742 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e5925e3f3c4b04b2e3d151ddb0b742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:32.030222 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.028522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.0.98.3:43003)
0924 04:41:32.028668 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:32.028674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.028719 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:41:32.028946 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.028949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.029010 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.029148 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:32.029783 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:32.029786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:32.029789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:32.029807 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:32.029825 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:32.029832 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:32.029924 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:32.030030 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:41:32.030075 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:32.036317 5166 raft_consensus.cc:1261] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 1 FOLLOWER]: Refusing update from remote peer 61e5925e3f3c4b04b2e3d151ddb0b742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:32.036327 5297 raft_consensus.cc:1261] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 1 FOLLOWER]: Refusing update from remote peer 61e5925e3f3c4b04b2e3d151ddb0b742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:32.037060 5499 consensus_queue.cc:1035] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c45ba20016d4dc8b04dd99d35bf76c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:32.037274 5501 consensus_queue.cc:1035] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1e5ff97c28496abda071390bc53979" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:41:32.041558 5213 tablet.cc:2369] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:32.045092 5506 mvcc.cc:205] Tried to move back new op lower bound from 7332761157773385728 to 7332761157668646912. Current Snapshot: MvccSnapshot[applied={T|T < 7332761157773385728}]
I20260924 04:41:32.052901 5022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:32.053098 5022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:32.055536 5022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a1a31ee49ef41e1abb445bc004fd67a: Sending DeleteTablet for 3 replicas of tablet 2c0d37a6fb7342ffb070345bbd812faf
I20260924 04:41:32.056339 5277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0d37a6fb7342ffb070345bbd812faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:32 UTC) from {username='slave'} at 127.0.0.1:34136
I20260924 04:41:32.056430 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0d37a6fb7342ffb070345bbd812faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:32 UTC) from {username='slave'} at 127.0.0.1:38698
I20260924 04:41:32.056434 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0d37a6fb7342ffb070345bbd812faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:32 UTC) from {username='slave'} at 127.0.0.1:35590
I20260924 04:41:32.056893 5516 tablet_replica.cc:330] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: stopping tablet replica
I20260924 04:41:32.057076 5515 tablet_replica.cc:330] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1: stopping tablet replica
I20260924 04:41:32.057312 5514 tablet_replica.cc:330] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: stopping tablet replica
I20260924 04:41:32.057510 5515 raft_consensus.cc:2229] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:32.057562 5514 raft_consensus.cc:2229] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:32.057662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5084
I20260924 04:41:32.057900 5514 raft_consensus.cc:2258] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:32.057900 5515 raft_consensus.cc:2258] T 2c0d37a6fb7342ffb070345bbd812faf P 1c45ba20016d4dc8b04dd99d35bf76c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:32.060011 5516 raft_consensus.cc:2229] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:32.060482 5516 raft_consensus.cc:2258] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:32.061609 5514 ts_tablet_manager.cc:1905] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:32.062110 5516 ts_tablet_manager.cc:1905] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:32.064581 5514 ts_tablet_manager.cc:1918] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:32.064703 5514 log.cc:1196] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/2c0d37a6fb7342ffb070345bbd812faf
I20260924 04:41:32.065047 5514 ts_tablet_manager.cc:1939] T 2c0d37a6fb7342ffb070345bbd812faf P 1a1e5ff97c28496abda071390bc53979: Deleting consensus metadata
I20260924 04:41:32.065035 5516 ts_tablet_manager.cc:1918] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:32.065258 5516 log.cc:1196] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/2c0d37a6fb7342ffb070345bbd812faf
I20260924 04:41:32.065618 5516 ts_tablet_manager.cc:1939] T 2c0d37a6fb7342ffb070345bbd812faf P 61e5925e3f3c4b04b2e3d151ddb0b742: Deleting consensus metadata
I20260924 04:41:32.066715 5009 catalog_manager.cc:4841] TS 1a1e5ff97c28496abda071390bc53979 (127.0.98.2:46861): tablet 2c0d37a6fb7342ffb070345bbd812faf (table test-table [id=9f79ed7c9cd64248a63c24f17ab7f7ad]) successfully deleted
I20260924 04:41:32.066716 5008 catalog_manager.cc:4841] TS 61e5925e3f3c4b04b2e3d151ddb0b742 (127.0.98.3:43003): tablet 2c0d37a6fb7342ffb070345bbd812faf (table test-table [id=9f79ed7c9cd64248a63c24f17ab7f7ad]) successfully deleted
W20260924 04:41:32.067340 5010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46151 (error 108)
I20260924 04:41:32.068148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5215
I20260924 04:41:32.068353 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.068054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.0.1:38702)
0924 04:41:32.068205 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:32.068275 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 04:41:32.069980 5010 catalog_manager.cc:4584] TS 1c45ba20016d4dc8b04dd99d35bf76c1 (127.0.98.1:46151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c0d37a6fb7342ffb070345bbd812faf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46151: connect: Connection refused (error 111)
I20260924 04:41:32.075603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5346
I20260924 04:41:32.080736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 4993
2026-09-24T04:41:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (986 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:41:32.102632 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:41:32.104233 392 webserver.cc:460] Webserver started at https://127.0.0.1:44633/ using document root <none> and password file <none>
I20260924 04:41:32.104377 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:41:32.104612 392 webserver.cc:460] Webserver started at http://127.0.0.1:36297/ using document root <none> and password file <none>
2026-09-24T04:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:32Z Disabled control of system clock
I20260924 04:41:32.113806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44633/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/testchainca.cert with env {}
W20260924 04:41:32.224622 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:32.224895 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:32.224934 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:32.229084 5523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:32.229147 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:32.229162 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:32.229180 5523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:32.229195 5523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:41:32.229211 5523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:41:32.229224 5523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44633/jwks.json
I20260924 04:41:32.234586 5523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44633/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:32.235716 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:32.236685 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:32.245224 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:32.245194 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:32.245234 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:32.245326 5523 server_base.cc:1034] running on GCE node
I20260924 04:41:32.246027 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:32.247258 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:32.248453 5523 hybrid_clock.cc:648] HybridClock initialized: now 1790224892248433 us; error 46 us; skew 500 ppm
I20260924 04:41:32.250402 5523 webserver.cc:460] Webserver started at http://127.0.98.62:45995/ using document root <none> and password file <none>
I20260924 04:41:32.251040 5523 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:32.251240 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:32.251493 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:32.253676 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "e13946fab9e44aa3b789656dfd39e6a9"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "0d85452a217056389325d7022a67c457"
server_key_iv: "c2d43b4b979258b12e70373d1afac780"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.254284 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "e13946fab9e44aa3b789656dfd39e6a9"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "0d85452a217056389325d7022a67c457"
server_key_iv: "c2d43b4b979258b12e70373d1afac780"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.258152 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 04:41:32.260836 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:32.262163 5523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:32.262297 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "e13946fab9e44aa3b789656dfd39e6a9"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "0d85452a217056389325d7022a67c457"
server_key_iv: "c2d43b4b979258b12e70373d1afac780"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.262418 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:32.285555 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:32.286315 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:32.286522 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:32.293373 5523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32815
I20260924 04:41:32.293387 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32815 every 8 connection(s)
I20260924 04:41:32.294339 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:32.297670 5590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:32.299769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5523
I20260924 04:41:32.299885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:32.300168 392 external_mini_cluster.cc:1428] Setting key 27af6f000b5a7c12b90ffd28004dee7d
I20260924 04:41:32.303820 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9: Bootstrap starting.
I20260924 04:41:32.306118 5590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:32.306829 5590 log.cc:824] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:32.308753 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9: No bootstrap required, opened a new log
I20260924 04:41:32.309702 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.301538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.98.62:32815)
0924 04:41:32.302188 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:32.302203 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.302239 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:32.302963 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.302980 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.303444 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.303783 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:32.304227 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.305237 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.306158 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.306676 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.306852 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.307890 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:32.307895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:32.307907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:32.308339 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:32.308394 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:32.308413 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:32.308520 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:32.308739 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:41:32.309031 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260924 04:41:32.312906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:41:32.312748 5590 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13946fab9e44aa3b789656dfd39e6a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260924 04:41:32.313004 5590 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:32.313056 5590 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13946fab9e44aa3b789656dfd39e6a9, State: Initialized, Role: FOLLOWER
I20260924 04:41:32.313530 5590 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13946fab9e44aa3b789656dfd39e6a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260924 04:41:32.313724 5590 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:32.313788 5590 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:32.313920 5590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:32.314994 5590 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13946fab9e44aa3b789656dfd39e6a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260924 04:41:32.315361 5590 leader_election.cc:302] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13946fab9e44aa3b789656dfd39e6a9; no voters:
I20260924 04:41:32.315683 5590 leader_election.cc:288] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:32.315826 5595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:32.316052 5595 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [term 1 LEADER]: Becoming Leader. State: Replica: e13946fab9e44aa3b789656dfd39e6a9, State: Running, Role: LEADER
I20260924 04:41:32.316395 5595 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13946fab9e44aa3b789656dfd39e6a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260924 04:41:32.316903 5590 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:32.318045 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13946fab9e44aa3b789656dfd39e6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13946fab9e44aa3b789656dfd39e6a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260924 04:41:32.318184 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:32.318511 5597 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13946fab9e44aa3b789656dfd39e6a9. Latest consensus state: current_term: 1 leader_uuid: "e13946fab9e44aa3b789656dfd39e6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13946fab9e44aa3b789656dfd39e6a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260924 04:41:32.318643 5597 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:32.318578 5604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:32.322165 5604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:32.327814 5604 catalog_manager.cc:1347] Generated new cluster ID: fa4cff1fd0a84bee95dfe4cd27d69dc4
I20260924 04:41:32.327889 5604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:32.337452 5604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:32.338356 5604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:32.353816 5604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13946fab9e44aa3b789656dfd39e6a9: Generated new TSK 0
I20260924 04:41:32.354611 5604 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:41:32.381632 5620 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:41:32.381757 5620 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:41:32.382385 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.365315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.98.62:32815)
0924 04:41:32.365460 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:32.365469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.365634 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:41:32.365877 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.365880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.365942 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.366182 (+ 240us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:41:32.366660 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.367286 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.368188 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.368418 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.368534 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.368866 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:41:32.381941 (+ 13075us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:41:32.382239 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:41:32.382459 5593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:41:32.383649 5619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:32.365168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.0.1:49680)
0924 04:41:32.365538 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:32.365552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:32.365692 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:32.366216 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:32.366220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:32.366228 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:41:32.366353 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:32.366359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.367398 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.367402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:32.368057 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:32.368065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.368517 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.368539 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:32.368631 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.368672 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:41:32.382174 (+ 13502us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:41:32.382414 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:32815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:41:32.384363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5523
2026-09-24T04:41:32Z chronyd exiting
I20260924 04:41:32.421778 392 test_util.cc:175] -----------------------------------------------
I20260924 04:41:32.421869 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224865297037-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:41:32.422971 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:41:32.423964 392 webserver.cc:460] Webserver started at https://127.0.0.1:36743/ using document root <none> and password file <none>
I20260924 04:41:32.424077 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:41:32.424355 392 webserver.cc:460] Webserver started at http://127.0.0.1:41835/ using document root <none> and password file <none>
2026-09-24T04:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:32Z Disabled control of system clock
I20260924 04:41:32.433859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45999
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45999
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/testchainca.cert with env {}
W20260924 04:41:32.547596 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:32.547878 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:32.547916 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:32.552057 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:32.552124 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:32.552140 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:32.552160 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:32.552176 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:41:32.552192 5626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:41:32.552205 5626 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36743/jwks.json
I20260924 04:41:32.557539 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:32.558657 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:32.559760 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:32.568876 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:32.568929 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:32.568940 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:32.569100 5626 server_base.cc:1034] running on GCE node
I20260924 04:41:32.569738 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:32.570928 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:32.572114 5626 hybrid_clock.cc:648] HybridClock initialized: now 1790224892572096 us; error 34 us; skew 500 ppm
I20260924 04:41:32.574139 5626 webserver.cc:460] Webserver started at http://127.0.98.62:41155/ using document root <none> and password file <none>
I20260924 04:41:32.574776 5626 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:32.574867 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:32.575098 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:32.577234 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "cc68127f0c514b6b9aa4a73c3674cf3e"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "8142ff8fca04f9bc1509080101f796ef"
server_key_iv: "3534f783dbcfc63f75cb1ea2f4367ce8"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.577765 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc68127f0c514b6b9aa4a73c3674cf3e"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "8142ff8fca04f9bc1509080101f796ef"
server_key_iv: "3534f783dbcfc63f75cb1ea2f4367ce8"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.581631 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:32.584342 5640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:32.585847 5626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:32.585955 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "cc68127f0c514b6b9aa4a73c3674cf3e"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "8142ff8fca04f9bc1509080101f796ef"
server_key_iv: "3534f783dbcfc63f75cb1ea2f4367ce8"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.586053 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:32.596725 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:32.597398 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:32.597546 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:32.604663 5626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45999
I20260924 04:41:32.604669 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45999 every 8 connection(s)
I20260924 04:41:32.605634 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:32.608896 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:32.609671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5626
I20260924 04:41:32.609771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:32.610070 392 external_mini_cluster.cc:1428] Setting key ab68d5a5e02ed3963f23222b2bddbcc5
I20260924 04:41:32.615760 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e: Bootstrap starting.
I20260924 04:41:32.618580 5693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:32.619586 5693 log.cc:824] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:32.621399 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.611569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.0.98.62:45999)
0924 04:41:32.612280 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:32.612299 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.612344 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:41:32.613223 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.613247 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.613787 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.614167 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:32.614636 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.616116 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.617061 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.617776 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.617940 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.618986 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:32.618993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:32.619010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:32.619578 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:32.619655 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:32.619733 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:32.619842 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:32.619977 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:41:32.620373 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":114,"threads_started":1}
I20260924 04:41:32.621996 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e: No bootstrap required, opened a new log
I20260924 04:41:32.624372 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:41:32.626466 5693 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45999 } }
I20260924 04:41:32.626715 5693 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:32.626775 5693 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc68127f0c514b6b9aa4a73c3674cf3e, State: Initialized, Role: FOLLOWER
I20260924 04:41:32.627285 5693 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45999 } }
I20260924 04:41:32.627434 5693 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:32.627502 5693 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:32.627601 5693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:32.628621 5693 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45999 } }
I20260924 04:41:32.628979 5693 leader_election.cc:302] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc68127f0c514b6b9aa4a73c3674cf3e; no voters:
I20260924 04:41:32.629333 5693 leader_election.cc:288] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:32.629504 5698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:32.629798 5698 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [term 1 LEADER]: Becoming Leader. State: Replica: cc68127f0c514b6b9aa4a73c3674cf3e, State: Running, Role: LEADER
I20260924 04:41:32.630106 5698 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45999 } }
I20260924 04:41:32.630712 5693 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:32.631914 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc68127f0c514b6b9aa4a73c3674cf3e. Latest consensus state: current_term: 1 leader_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45999 } } }
I20260924 04:41:32.632042 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:32.632539 5707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:32.633095 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc68127f0c514b6b9aa4a73c3674cf3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45999 } } }
I20260924 04:41:32.633221 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:32.636757 5707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:32.643167 5707 catalog_manager.cc:1347] Generated new cluster ID: b879de1cb18540469a2545ba46d4bc0f
I20260924 04:41:32.643252 5707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:32.680496 5707 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:32.681836 5707 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:32.687027 5707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc68127f0c514b6b9aa4a73c3674cf3e: Generated new TSK 0
I20260924 04:41:32.687804 5707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:41:32.719092 5723 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:41:32.719282 5723 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:41:32.720126 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.703080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.98.62:45999)
0924 04:41:32.703260 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:32.703265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.703423 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:41:32.703581 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.703584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.703650 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.703785 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:41:32.704064 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.704930 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.705860 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.706219 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.706325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.706641 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:41:32.719613 (+ 12972us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:41:32.719978 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 04:41:32.720199 5696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:41:32.720256 5722 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:32.702950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45999 (local address 127.0.0.1:44146)
0924 04:41:32.703326 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:32.703339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:32.703448 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:32.703816 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:32.703819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:32.703827 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:41:32.703926 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:32.703931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.705087 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.705090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:32.705760 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:32.705768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.706327 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.706346 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:32.706412 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.706436 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:41:32.719878 (+ 13442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:41:32.720050 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260924 04:41:32.721266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5626
2026-09-24T04:41:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (337 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:41:32.760810 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:41:32.761663 392 webserver.cc:460] Webserver started at https://127.0.0.1:38121/ using document root <none> and password file <none>
I20260924 04:41:32.761747 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:41:32.761986 392 webserver.cc:460] Webserver started at http://127.0.0.1:43967/ using document root <none> and password file <none>
2026-09-24T04:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:32Z Disabled control of system clock
I20260924 04:41:32.770869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39825
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39825
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38121/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:41:32.884634 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:32.884958 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:32.885000 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:32.889122 5729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:32.889199 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:32.889215 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:32.889233 5729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:32.889250 5729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:41:32.889266 5729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:41:32.889281 5729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38121/jwks.json
I20260924 04:41:32.894289 5729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38121/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:32.895609 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:32.897159 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:41:32.906620 5729 server_base.cc:1034] running on GCE node
W20260924 04:41:32.906669 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:32.906622 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:32.906876 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:32.907649 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:32.908841 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:32.910017 5729 hybrid_clock.cc:648] HybridClock initialized: now 1790224892909999 us; error 32 us; skew 500 ppm
I20260924 04:41:32.912079 5729 webserver.cc:460] Webserver started at http://127.0.98.62:33543/ using document root <none> and password file <none>
I20260924 04:41:32.912700 5729 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:32.912766 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:32.913012 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:32.915239 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "840c6b16e762474fa4c56f882127dce4"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "39cd87ef4c98410c389167c753700ef0"
server_key_iv: "db2e43fce79c7daeb2709c61894cef28"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.915853 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "840c6b16e762474fa4c56f882127dce4"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "39cd87ef4c98410c389167c753700ef0"
server_key_iv: "db2e43fce79c7daeb2709c61894cef28"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.920305 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 04:41:32.922888 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:32.924238 5729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:41:32.924388 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "840c6b16e762474fa4c56f882127dce4"
format_stamp: "Formatted at 2026-09-24 04:41:32 on dist-test-slave-npg2"
server_key: "39cd87ef4c98410c389167c753700ef0"
server_key_iv: "db2e43fce79c7daeb2709c61894cef28"
server_key_version: "encryptionkey@0"
I20260924 04:41:32.924525 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:32.967242 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:32.968051 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:32.968277 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:32.975920 5729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39825
I20260924 04:41:32.975945 5795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39825 every 8 connection(s)
I20260924 04:41:32.976955 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:32.977274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5729
I20260924 04:41:32.977370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:32.977720 392 external_mini_cluster.cc:1428] Setting key 13e7adc566b26b2612bb4ded795a24da
I20260924 04:41:32.980973 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:32.987509 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4: Bootstrap starting.
I20260924 04:41:32.987607 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:32.979302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.0.98.62:39825)
0924 04:41:32.980033 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:32.980048 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:32.980087 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:41:32.980780 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:32.980797 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:32.981277 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:32.981683 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:32.982135 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.983171 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.984111 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:32.984607 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:32.984787 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:32.985717 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:32.985722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:32.985734 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:32.986154 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:32.986212 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:32.986232 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:32.986345 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:32.986485 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:41:32.986763 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":144,"threads_started":1}
I20260924 04:41:32.990195 5796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:32.990590 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:41:32.991161 5796 log.cc:824] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:32.993561 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4: No bootstrap required, opened a new log
I20260924 04:41:32.998927 5796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c6b16e762474fa4c56f882127dce4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 04:41:32.999281 5796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:32.999370 5796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840c6b16e762474fa4c56f882127dce4, State: Initialized, Role: FOLLOWER
I20260924 04:41:32.999871 5796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c6b16e762474fa4c56f882127dce4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 04:41:33.000032 5796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:33.000125 5796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:33.000233 5796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:33.094413 5796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c6b16e762474fa4c56f882127dce4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 04:41:33.095000 5796 leader_election.cc:302] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840c6b16e762474fa4c56f882127dce4; no voters:
I20260924 04:41:33.095515 5796 leader_election.cc:288] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:33.095717 5801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:33.095980 5801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [term 1 LEADER]: Becoming Leader. State: Replica: 840c6b16e762474fa4c56f882127dce4, State: Running, Role: LEADER
I20260924 04:41:33.096336 5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c6b16e762474fa4c56f882127dce4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 04:41:33.097147 5796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:33.098152 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "840c6b16e762474fa4c56f882127dce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c6b16e762474fa4c56f882127dce4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } } }
I20260924 04:41:33.098284 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:33.098670 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840c6b16e762474fa4c56f882127dce4. Latest consensus state: current_term: 1 leader_uuid: "840c6b16e762474fa4c56f882127dce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c6b16e762474fa4c56f882127dce4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } } }
I20260924 04:41:33.098843 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:33.098757 5808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:33.102406 5808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:33.107906 5808 catalog_manager.cc:1347] Generated new cluster ID: e85e63732b004d7a94e42baddbfdf554
I20260924 04:41:33.107985 5808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:33.121960 5808 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:33.123032 5808 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:33.128242 5808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 840c6b16e762474fa4c56f882127dce4: Generated new TSK 0
I20260924 04:41:33.129168 5808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:41:33.144995 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:33.139648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.0.98.62:39825)
0924 04:41:33.139786 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:33.139791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:33.139944 (+ 153us) server_negotiation.cc:407] Connection header received
0924 04:41:33.140111 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:33.140115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:33.140178 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:33.140311 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:41:33.140523 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.141372 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.142248 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.142593 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.142697 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:33.143016 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:41:33.144604 (+ 1588us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38121/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:41:33.144846 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55380: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38121/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:41:33.145077 5799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55380: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38121/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 04:41:33.146678 5825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:33.139531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39825 (local address 127.0.0.1:55380)
0924 04:41:33.139849 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:33.139870 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:33.139971 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:33.140313 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:33.140316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:33.140322 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:41:33.140419 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:33.140423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.141483 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.141485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:33.142112 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:33.142119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.142723 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.142744 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:33.142811 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:33.142835 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:41:33.144776 (+ 1941us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38121/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:41:33.146496 (+ 1720us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39825: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38121/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1442,"thread_start_us":111,"threads_started":1}
I20260924 04:41:33.147303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5729
E20260924 04:41:33.157721 5826 webserver.cc:533] Webserver: sslize failed:
E20260924 04:41:33.157814 5826 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:41:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (427 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:33Z Disabled control of system clock
I20260924 04:41:33.197157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38867
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38867
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/testchainca.cert with env {}
W20260924 04:41:33.315340 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:33.315649 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:33.315693 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:33.321108 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:33.321198 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:33.321215 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:33.321235 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:33.321252 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:41:33.321323 5830 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:41:33.321358 5830 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:41:33.327188 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:33.328341 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:33.329422 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:33.338370 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:33.338366 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:33.338366 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:33.338671 5830 server_base.cc:1034] running on GCE node
I20260924 04:41:33.339380 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:33.340562 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:33.341759 5830 hybrid_clock.cc:648] HybridClock initialized: now 1790224893341736 us; error 35 us; skew 500 ppm
I20260924 04:41:33.343999 5830 webserver.cc:460] Webserver started at http://127.0.98.62:44783/ using document root <none> and password file <none>
I20260924 04:41:33.344611 5830 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:33.344672 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:33.344833 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:33.347666 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "a3995e7a2daa4a2ea833e74a931a8867"
format_stamp: "Formatted at 2026-09-24 04:41:33 on dist-test-slave-npg2"
server_key: "44fd4e280d45d30187e796893d37242e"
server_key_iv: "685d563d3f0a07bf104aae0ccb58f978"
server_key_version: "encryptionkey@0"
I20260924 04:41:33.348192 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3995e7a2daa4a2ea833e74a931a8867"
format_stamp: "Formatted at 2026-09-24 04:41:33 on dist-test-slave-npg2"
server_key: "44fd4e280d45d30187e796893d37242e"
server_key_iv: "685d563d3f0a07bf104aae0ccb58f978"
server_key_version: "encryptionkey@0"
I20260924 04:41:33.352190 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:41:33.354732 5844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:33.356125 5830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:33.356276 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "a3995e7a2daa4a2ea833e74a931a8867"
format_stamp: "Formatted at 2026-09-24 04:41:33 on dist-test-slave-npg2"
server_key: "44fd4e280d45d30187e796893d37242e"
server_key_iv: "685d563d3f0a07bf104aae0ccb58f978"
server_key_version: "encryptionkey@0"
I20260924 04:41:33.356400 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:33.368311 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:33.369094 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:33.369303 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:33.377282 5830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38867
I20260924 04:41:33.377286 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38867 every 8 connection(s)
I20260924 04:41:33.378376 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:33.381755 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:33.383496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5830
I20260924 04:41:33.383626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:33.383925 392 external_mini_cluster.cc:1428] Setting key 6ed76402276ff92badcdbca3171d0e04
I20260924 04:41:33.389603 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867: Bootstrap starting.
I20260924 04:41:33.392316 5897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:33.393106 5897 log.cc:824] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:33.394588 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:33.385559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.0.98.62:38867)
0924 04:41:33.386202 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:33.386218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:33.386256 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:41:33.386935 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:33.386954 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:33.387489 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:33.387859 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:33.388346 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.389918 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.390951 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.391515 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.391717 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:33.392643 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:33.392649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:33.392663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:41:33.393100 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:33.393158 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:41:33.393178 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:33.393292 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:33.393412 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 04:41:33.393689 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
I20260924 04:41:33.395529 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867: No bootstrap required, opened a new log
I20260924 04:41:33.397891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:41:33.400243 5897 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3995e7a2daa4a2ea833e74a931a8867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260924 04:41:33.400485 5897 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:33.400530 5897 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3995e7a2daa4a2ea833e74a931a8867, State: Initialized, Role: FOLLOWER
I20260924 04:41:33.400983 5897 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3995e7a2daa4a2ea833e74a931a8867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260924 04:41:33.401117 5897 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:33.401160 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:33.401295 5897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:33.489480 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3995e7a2daa4a2ea833e74a931a8867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260924 04:41:33.490103 5897 leader_election.cc:302] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3995e7a2daa4a2ea833e74a931a8867; no voters:
I20260924 04:41:33.490576 5897 leader_election.cc:288] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:33.490769 5902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:33.491046 5902 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [term 1 LEADER]: Becoming Leader. State: Replica: a3995e7a2daa4a2ea833e74a931a8867, State: Running, Role: LEADER
I20260924 04:41:33.491441 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3995e7a2daa4a2ea833e74a931a8867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260924 04:41:33.492234 5897 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:33.493221 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3995e7a2daa4a2ea833e74a931a8867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3995e7a2daa4a2ea833e74a931a8867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } } }
I20260924 04:41:33.493355 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:33.493443 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3995e7a2daa4a2ea833e74a931a8867. Latest consensus state: current_term: 1 leader_uuid: "a3995e7a2daa4a2ea833e74a931a8867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3995e7a2daa4a2ea833e74a931a8867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } } }
I20260924 04:41:33.493558 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:33.493929 5910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:33.497982 5910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:33.504509 5910 catalog_manager.cc:1347] Generated new cluster ID: 627081f909c647b2b3b77eeeececdcff
I20260924 04:41:33.504652 5910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:33.514631 5910 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:33.515708 5910 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:33.530663 5910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3995e7a2daa4a2ea833e74a931a8867: Generated new TSK 0
I20260924 04:41:33.531556 5910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:41:33.607931 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:33.543209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.98.62:38867)
0924 04:41:33.543357 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:33.543361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:33.543515 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:41:33.543750 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:33.543753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:33.543816 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:33.543947 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:41:33.544164 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.545015 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.545954 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.546334 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.546438 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:33.546745 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:41:33.607437 (+ 60692us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:41:33.607780 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54574: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:41:33.608029 5900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54574: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:41:33.608119 5926 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:33.543059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38867 (local address 127.0.0.1:54574)
0924 04:41:33.543430 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:33.543443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:33.543610 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:33.543949 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:33.543951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:33.543958 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:41:33.544057 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:33.544062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.545144 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.545147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:33.545834 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:33.545842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:33.546444 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:33.546467 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:33.546535 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:33.546558 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:41:33.607699 (+ 61141us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:41:33.607902 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260924 04:41:33.608899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5830
2026-09-24T04:41:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (440 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (129 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:34 dist-test-slave-npg2 krb5kdc[5938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:34 dist-test-slave-npg2 krb5kdc[5938](info): set up 2 sockets
Sep 24 04:41:34 dist-test-slave-npg2 krb5kdc[5938](info): commencing operation
krb5kdc: starting...
W20260924 04:41:35.067834 392 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:35.194929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:35.314105 5951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:35.314414 5951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:35.314458 5951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:35.318773 5951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:35.318845 5951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:35.318864 5951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:35.318881 5951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:35.318897 5951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:35.324433 5951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:35.325496 5951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:35.326494 5951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:35.335160 5957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:35.335162 5956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:35.335254 5959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:35.335307 5951 server_base.cc:1034] running on GCE node
I20260924 04:41:35.336016 5951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:35.337111 5951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:35.338310 5951 hybrid_clock.cc:648] HybridClock initialized: now 1790224895338301 us; error 33 us; skew 500 ppm
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:35.346028 5951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:35.346961 5951 webserver.cc:460] Webserver started at http://127.0.98.62:46659/ using document root <none> and password file <none>
I20260924 04:41:35.347599 5951 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:35.347661 5951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:35.347887 5951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:35.349974 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "29d81cd128d343e4b019f5a07352f2c5"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "dceb6296210dc32bd310204469699076"
server_key_iv: "5b9fe6187ad6fec49e9fa503bb209698"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.350555 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "29d81cd128d343e4b019f5a07352f2c5"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "dceb6296210dc32bd310204469699076"
server_key_iv: "5b9fe6187ad6fec49e9fa503bb209698"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.354468 5951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:41:35.357004 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.358299 5951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:41:35.358429 5951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "29d81cd128d343e4b019f5a07352f2c5"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "dceb6296210dc32bd310204469699076"
server_key_iv: "5b9fe6187ad6fec49e9fa503bb209698"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.358548 5951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:35.390381 5951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:35.393648 5951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:35.393862 5951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:35.401288 5951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44451
I20260924 04:41:35.401619 6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44451 every 8 connection(s)
I20260924 04:41:35.402299 5951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:35.405176 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:35.410535 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5: Bootstrap starting.
I20260924 04:41:35.411952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 5951
I20260924 04:41:35.412082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:35.412369 392 external_mini_cluster.cc:1428] Setting key f6c148bc0b27e901f93a0a6e4343ba5c
I20260924 04:41:35.413543 6019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:35.414304 6019 log.cc:824] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:35.416594 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5: No bootstrap required, opened a new log
I20260924 04:41:35.421012 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d81cd128d343e4b019f5a07352f2c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44451 } }
I20260924 04:41:35.421346 6019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:35.421418 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d81cd128d343e4b019f5a07352f2c5, State: Initialized, Role: FOLLOWER
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:35.421942 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d81cd128d343e4b019f5a07352f2c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44451 } }
I20260924 04:41:35.422147 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:35.422216 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:35.422327 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:35.423353 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d81cd128d343e4b019f5a07352f2c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44451 } }
I20260924 04:41:35.423734 6019 leader_election.cc:302] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d81cd128d343e4b019f5a07352f2c5; no voters:
I20260924 04:41:35.424150 6019 leader_election.cc:288] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:35.424263 6024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:35.424474 6024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [term 1 LEADER]: Becoming Leader. State: Replica: 29d81cd128d343e4b019f5a07352f2c5, State: Running, Role: LEADER
I20260924 04:41:35.424888 6024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d81cd128d343e4b019f5a07352f2c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44451 } }
I20260924 04:41:35.425565 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:35.426852 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d81cd128d343e4b019f5a07352f2c5. Latest consensus state: current_term: 1 leader_uuid: "29d81cd128d343e4b019f5a07352f2c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d81cd128d343e4b019f5a07352f2c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44451 } } }
I20260924 04:41:35.426975 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:35.427419 6033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:35.427467 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d81cd128d343e4b019f5a07352f2c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d81cd128d343e4b019f5a07352f2c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44451 } } }
I20260924 04:41:35.427598 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:35.429374 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:35.413507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.98.62:44451)
0924 04:41:35.413954 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:35.413970 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:35.414006 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:35.414755 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:35.414780 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:35.415247 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:35.415531 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:35.416389 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.417459 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.418681 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.419426 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.419578 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:35.422719 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:35.422724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:35.422741 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:35.422771 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:35.425068 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:35.425659 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:35.425665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:35.425671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:35.425822 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:35.426187 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:35.426190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:35.426192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:35.426754 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:35.426948 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:35.427877 (+ 929us) server_negotiation.cc:299] Negotiation successful
0924 04:41:35.428275 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260924 04:41:35.430914 6033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:35.436749 6033 catalog_manager.cc:1347] Generated new cluster ID: 3d4e2fed6a5a45ab96fd141ca640449a
I20260924 04:41:35.436836 6033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:35.445747 6033 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:35.446735 6033 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:35.451043 6033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d81cd128d343e4b019f5a07352f2c5: Generated new TSK 0
I20260924 04:41:35.451861 6033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:35.489981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44451
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:35.623411 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:35.623682 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:35.623726 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:35.628324 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:35.628388 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:35.628475 6045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:35.634286 6045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:35.635438 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:35.636536 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:35.646341 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:35.646301 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:35.646306 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:35.646564 6045 server_base.cc:1034] running on GCE node
I20260924 04:41:35.647189 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:35.647845 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:35.649012 6045 hybrid_clock.cc:648] HybridClock initialized: now 1790224895648981 us; error 47 us; skew 500 ppm
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:35.652026 6045 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:35.652889 6045 webserver.cc:460] Webserver started at http://127.0.98.1:42315/ using document root <none> and password file <none>
I20260924 04:41:35.653443 6045 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:35.653501 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:35.653710 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:35.655772 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "d94396d04da246cab92967d6626502e8"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "877675f74334972fdf0329a4c27fd457"
server_key_iv: "03c490325bc314a703f9f17369a36a26"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.656312 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d94396d04da246cab92967d6626502e8"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "877675f74334972fdf0329a4c27fd457"
server_key_iv: "03c490325bc314a703f9f17369a36a26"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.660043 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:35.662673 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.664053 6045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:35.664222 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "d94396d04da246cab92967d6626502e8"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "877675f74334972fdf0329a4c27fd457"
server_key_iv: "03c490325bc314a703f9f17369a36a26"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.664323 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:35.682807 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:35.685884 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:35.686096 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:35.686707 6045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:35.687599 6045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:35.687678 6045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.687750 6045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:35.687798 6045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.696843 6045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41195
I20260924 04:41:35.696866 6173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41195 every 8 connection(s)
I20260924 04:41:35.697742 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:35.707495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6045
I20260924 04:41:35.707620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:35.707923 392 external_mini_cluster.cc:1428] Setting key ad5c5fdd691ebd05f529038ee855fe7d
I20260924 04:41:35.712105 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:35.699675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53727 (local address 127.0.98.62:44451)
0924 04:41:35.699863 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:35.699868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:35.700783 (+ 915us) server_negotiation.cc:407] Connection header received
0924 04:41:35.702132 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:35.702135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:35.702180 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:35.702258 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:35.703440 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.703987 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.705182 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.705426 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.705541 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:35.708215 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:35.708219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:35.708222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:35.708250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:35.709902 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:35.710441 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:35.710447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:35.710452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:35.710544 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:35.710860 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:35.710866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:35.710871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:35.711117 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:35.711250 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:35.711890 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 04:41:35.711988 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 04:41:35.713377 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:35.700188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44451 (local address 127.0.98.1:53727)
0924 04:41:35.700622 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:35.700664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:35.701920 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:35.702427 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:35.702436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:35.702788 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:35.703305 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:35.703319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.704129 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.704133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:35.705031 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:35.705041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.705583 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.705587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:35.705755 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:35.706322 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:35.706337 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:35.708052 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:35.710020 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:35.710024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:35.710037 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:35.710342 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:35.710636 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:35.710638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:35.710639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:35.710771 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:35.711276 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:35.711286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:35.711628 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 04:41:35.711852 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 04:41:35.712248 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260924 04:41:35.714752 6174 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44451
I20260924 04:41:35.715008 6174 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:35.715550 6174 heartbeater.cc:511] Master 127.0.98.62:44451 requested a full tablet report, sending...
I20260924 04:41:35.717173 5983 ts_manager.cc:194] Registered new tserver with Master: d94396d04da246cab92967d6626502e8 (127.0.98.1:41195)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:35.718350 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:35.738014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44451
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:35.850867 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:35.851239 6179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:35.851329 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:35.855482 6179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:35.855561 6179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:35.855645 6179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:35.861056 6179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:35.862231 6179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:35.863484 6179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:35.873203 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:35.873296 6179 server_base.cc:1034] running on GCE node
W20260924 04:41:35.873232 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:35.873224 6187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:35.873955 6179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:35.874572 6179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:35.875775 6179 hybrid_clock.cc:648] HybridClock initialized: now 1790224895875758 us; error 31 us; skew 500 ppm
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:35.878618 6179 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:35.879639 6179 webserver.cc:460] Webserver started at http://127.0.98.2:34095/ using document root <none> and password file <none>
I20260924 04:41:35.880209 6179 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:35.880295 6179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:35.880518 6179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:35.882553 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "5bf01e1f76f6404b8ae8cc7da2f9253a"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "b1523abbdcd771252f9350594b73a4d2"
server_key_iv: "2e0f389a95f680285214b48fac3ab92c"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.883204 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5bf01e1f76f6404b8ae8cc7da2f9253a"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "b1523abbdcd771252f9350594b73a4d2"
server_key_iv: "2e0f389a95f680285214b48fac3ab92c"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.886963 6179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:35.889626 6194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.890913 6179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:35.891040 6179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "5bf01e1f76f6404b8ae8cc7da2f9253a"
format_stamp: "Formatted at 2026-09-24 04:41:35 on dist-test-slave-npg2"
server_key: "b1523abbdcd771252f9350594b73a4d2"
server_key_iv: "2e0f389a95f680285214b48fac3ab92c"
server_key_version: "encryptionkey@0"
I20260924 04:41:35.891202 6179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:35.909768 6179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:35.913096 6179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:35.913311 6179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:35.913897 6179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:35.914716 6179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:35.914762 6179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.914801 6179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:35.914855 6179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:35.924331 6179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34141
I20260924 04:41:35.924403 6307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34141 every 8 connection(s)
I20260924 04:41:35.925292 6179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:35.934681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6179
I20260924 04:41:35.934823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:35.935101 392 external_mini_cluster.cc:1428] Setting key 9b781091f6fd5b0f05b97a7361598ef8
Sep 24 04:41:35 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:35.940052 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:35.927347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39727 (local address 127.0.98.62:44451)
0924 04:41:35.927515 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:35.927521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:35.928331 (+ 810us) server_negotiation.cc:407] Connection header received
0924 04:41:35.929759 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:35.929762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:35.929809 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:35.929893 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:35.931071 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.931866 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.932974 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.933221 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.933331 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:35.936091 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:35.936094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:35.936096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:35.936138 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:35.937812 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:35.938326 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:35.938332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:35.938336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:35.938438 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:35.938867 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:35.938870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:35.938871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:35.939048 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:35.939188 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:35.939832 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 04:41:35.939930 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:41:35.941350 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:35.927714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44451 (local address 127.0.98.2:39727)
0924 04:41:35.928171 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:35.928224 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:35.929535 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:35.930052 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:35.930058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:35.930409 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:35.930942 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:35.930956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.931974 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.931977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:35.932846 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:35.932854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:35.933405 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:35.933408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:35.933565 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:35.934140 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:35.934155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:35.935938 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:35.937936 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:35.937941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:35.937953 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:35.938219 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:35.938556 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:35.938561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:35.938564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:35.938768 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:35.939166 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:35.939174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:35.939524 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 04:41:35.939805 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 04:41:35.940180 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":136,"threads_started":1}
I20260924 04:41:35.942737 6308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44451
I20260924 04:41:35.942992 6308 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:35.943616 6308 heartbeater.cc:511] Master 127.0.98.62:44451 requested a full tablet report, sending...
I20260924 04:41:35.944692 5983 ts_manager.cc:194] Registered new tserver with Master: 5bf01e1f76f6404b8ae8cc7da2f9253a (127.0.98.2:34141)
I20260924 04:41:35.945282 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39727
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:35.966666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44451
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:36.081112 6313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:36.081460 6313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:36.081542 6313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:36.085671 6313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:36.085778 6313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:36.085879 6313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:36.091311 6313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:36.092593 6313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:36.093715 6313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:36.103636 6321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:36.103636 6318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:36.103658 6319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:36.103938 6313 server_base.cc:1034] running on GCE node
I20260924 04:41:36.104417 6313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:36.105068 6313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:36.106274 6313 hybrid_clock.cc:648] HybridClock initialized: now 1790224896106258 us; error 34 us; skew 500 ppm
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.109123 6313 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:36.110013 6313 webserver.cc:460] Webserver started at http://127.0.98.3:41357/ using document root <none> and password file <none>
I20260924 04:41:36.110589 6313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:36.110682 6313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:36.110908 6313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:36.113142 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "77024b24d5904b2b814b8e559cceafcd"
format_stamp: "Formatted at 2026-09-24 04:41:36 on dist-test-slave-npg2"
server_key: "07122cc2c72162e39c783d4d306db32b"
server_key_iv: "eb76181aec1bc59c4e75cd92f22a1c4e"
server_key_version: "encryptionkey@0"
I20260924 04:41:36.113741 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77024b24d5904b2b814b8e559cceafcd"
format_stamp: "Formatted at 2026-09-24 04:41:36 on dist-test-slave-npg2"
server_key: "07122cc2c72162e39c783d4d306db32b"
server_key_iv: "eb76181aec1bc59c4e75cd92f22a1c4e"
server_key_version: "encryptionkey@0"
I20260924 04:41:36.117590 6313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:36.120360 6328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:36.121747 6313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:36.121901 6313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "77024b24d5904b2b814b8e559cceafcd"
format_stamp: "Formatted at 2026-09-24 04:41:36 on dist-test-slave-npg2"
server_key: "07122cc2c72162e39c783d4d306db32b"
server_key_iv: "eb76181aec1bc59c4e75cd92f22a1c4e"
server_key_version: "encryptionkey@0"
I20260924 04:41:36.122026 6313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:36.155241 6313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:36.158614 6313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:36.158838 6313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:36.159487 6313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:36.160351 6313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:36.160424 6313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:36.160495 6313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:36.160557 6313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:36.170121 6313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45761
I20260924 04:41:36.170147 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45761 every 8 connection(s)
I20260924 04:41:36.171044 6313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:36.173857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6313
I20260924 04:41:36.173992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:36.174341 392 external_mini_cluster.cc:1428] Setting key 2d3806e8ed0b48c9b65217671a479901
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:36.185551 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:36.172884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46157 (local address 127.0.98.62:44451)
0924 04:41:36.173026 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.173030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.173957 (+ 927us) server_negotiation.cc:407] Connection header received
0924 04:41:36.175955 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.175958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.176008 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.176115 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.177522 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.178037 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.179360 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.179569 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.179647 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:36.182081 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:36.182084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:36.182086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:36.182112 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:36.183624 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.184102 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.184105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.184106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.184186 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.184480 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.184483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.184484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.184654 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:36.184729 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:36.185316 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 04:41:36.185427 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:41:36.186367 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:36.173282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44451 (local address 127.0.98.3:46157)
0924 04:41:36.173784 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:36.173849 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:36.175726 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:36.176307 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:36.176314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:36.176882 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:36.177387 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:36.177400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.178164 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.178167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:36.179251 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:36.179261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.179701 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.179704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:36.179859 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:36.180417 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:36.180434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:36.181900 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:36.183739 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:36.183745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:36.183759 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:36.184010 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:36.184267 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:36.184270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:36.184271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:36.184401 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:36.184772 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:36.184778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:36.185019 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:41:36.185229 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 04:41:36.185497 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260924 04:41:36.187825 6442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44451
I20260924 04:41:36.188119 6442 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:36.188673 6442 heartbeater.cc:511] Master 127.0.98.62:44451 requested a full tablet report, sending...
I20260924 04:41:36.189728 5983 ts_manager.cc:194] Registered new tserver with Master: 77024b24d5904b2b814b8e559cceafcd (127.0.98.3:45761)
I20260924 04:41:36.190260 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46157
I20260924 04:41:36.200582 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.216521 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:41:36.219647 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc' of type FILE
W20260924 04:41:36.219580 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:36.217940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.98.1:41195)
0924 04:41:36.218277 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.218283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.218308 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:41:36.218360 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.218364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.218576 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.218776 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.219424 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42058: BlockingRecv error: recv got EOF from 127.0.0.1:42058 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.220836 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.221910 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:36.232297 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:36.222922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.0.98.1:41195)
0924 04:41:36.223073 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.223078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.223243 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:41:36.223406 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.223414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.223456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.223541 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.224092 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.224883 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.225786 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.226337 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.226452 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:36.229046 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:36.229054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:36.229068 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:36.229091 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:36.230654 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.231042 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.231047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.231049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.231168 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.231441 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.231443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.231445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.231676 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:36.231832 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:36.232036 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 04:41:36.232155 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 04:41:36.236943 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:36.235775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.98.1:41195)
0924 04:41:36.235896 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.235900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.236124 (+ 224us) server_negotiation.cc:407] Connection header received
0924 04:41:36.236303 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.236305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.236342 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.236425 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.236833 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42076: BlockingRecv error: recv got EOF from 127.0.0.1:42076 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:41:36.237112 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.238144 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.239085 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:36.248144 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:36.239934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.98.1:41195)
0924 04:41:36.240042 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.240046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.240227 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:41:36.240312 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.240314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.240351 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.240416 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.240925 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.241427 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.242237 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.242454 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.242535 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:36.245177 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:36.245180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:36.245182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:36.245204 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:36.246520 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.246947 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.246949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.246951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.247025 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.247363 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.247366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.247367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.247530 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:36.247740 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:36.247925 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 04:41:36.248024 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:41:36.250993 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:36.250046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.0.98.1:41195)
0924 04:41:36.250173 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.250176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.250189 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:41:36.250305 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.250307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.250341 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.250403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.250896 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42082: BlockingRecv error: recv got EOF from 127.0.0.1:42082 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:41:36.251165 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224865297037-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.252298 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:36.253237 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:41:36.262059 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:36.254173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.0.98.1:41195)
0924 04:41:36.254409 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:36.254413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:36.254424 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:41:36.254454 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:36.254456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:36.254488 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:36.254550 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:36.255022 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.255592 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.256368 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:36.256566 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:36.256639 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:36.258698 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:36.258701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:36.258704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:36.258730 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:36.260235 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.260809 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.260812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.260813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.260890 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:36.261171 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:36.261175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:36.261176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:36.261343 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:36.261499 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:36.261787 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 04:41:36.261911 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260924 04:41:36.263293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6045
I20260924 04:41:36.269342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6179
I20260924 04:41:36.274799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6313
I20260924 04:41:36.280366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 5951
2026-09-24T04:41:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2413 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[6462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[6462](info): set up 2 sockets
Sep 24 04:41:36 dist-test-slave-npg2 krb5kdc[6462](info): commencing operation
krb5kdc: starting...
W20260924 04:41:37.331779 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:37 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:41:37.462610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38757
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:37.578666 6475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:37.578953 6475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:37.578994 6475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:37.583290 6475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:37.583367 6475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:37.583387 6475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:37.583405 6475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:37.583419 6475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:37.589057 6475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:37.590189 6475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:37.591306 6475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:37.600116 6480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:37.600088 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:37.600082 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:37.600412 6475 server_base.cc:1034] running on GCE node
I20260924 04:41:37.601135 6475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:37.602427 6475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:37.603598 6475 hybrid_clock.cc:648] HybridClock initialized: now 1790224897603544 us; error 137 us; skew 500 ppm
Sep 24 04:41:37 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:37.606846 6475 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:41:37.607862 6475 webserver.cc:460] Webserver started at http://127.0.98.62:42035/ using document root <none> and password file <none>
I20260924 04:41:37.608415 6475 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:37.608475 6475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:37.608646 6475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:37.610770 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "6bd3e0098d31455d999a95d5809cefa8"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "f725e4d5b1c23fb82a018efb7a3b5b1c"
server_key_iv: "d5b5f15ebf0a2a82738c68fa3a53c29e"
server_key_version: "encryptionkey@0"
I20260924 04:41:37.611357 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bd3e0098d31455d999a95d5809cefa8"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "f725e4d5b1c23fb82a018efb7a3b5b1c"
server_key_iv: "d5b5f15ebf0a2a82738c68fa3a53c29e"
server_key_version: "encryptionkey@0"
I20260924 04:41:37.615504 6475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:41:37.618182 6490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:37.619640 6475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:37.619831 6475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "6bd3e0098d31455d999a95d5809cefa8"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "f725e4d5b1c23fb82a018efb7a3b5b1c"
server_key_iv: "d5b5f15ebf0a2a82738c68fa3a53c29e"
server_key_version: "encryptionkey@0"
I20260924 04:41:37.619962 6475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:37.637053 6475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:37.640622 6475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:37.640852 6475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:37.648474 6475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38757
I20260924 04:41:37.648855 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38757 every 8 connection(s)
I20260924 04:41:37.649564 6475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:37.650007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6475
I20260924 04:41:37.650148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:37.650449 392 external_mini_cluster.cc:1428] Setting key dd0fceff9be81592002ba4d150117136
I20260924 04:41:37.652786 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:37.659628 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Bootstrap starting.
Sep 24 04:41:37 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:37.661966 6543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:37.662710 6543 log.cc:824] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:37.664793 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: No bootstrap required, opened a new log
I20260924 04:41:37.665841 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:37.652196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.0.98.62:38757)
0924 04:41:37.652649 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:37.652669 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:37.652705 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:37.653405 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:37.653423 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:37.653779 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:37.654071 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:37.654984 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:37.656158 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:37.657300 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:37.657774 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:37.657901 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:37.660849 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:37.660854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:37.660868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:37.660896 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:37.662799 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:37.663367 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:37.663373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:37.663384 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:37.663497 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:37.663824 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:37.663827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:37.663829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:37.664244 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:37.664450 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:37.664749 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:41:37.665091 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":147,"threads_started":1}
I20260924 04:41:37.668924 6543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260924 04:41:37.669178 6543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:37.669260 6543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd3e0098d31455d999a95d5809cefa8, State: Initialized, Role: FOLLOWER
I20260924 04:41:37.669731 6543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260924 04:41:37.669878 6543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:37.669946 6543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:37.670042 6543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:37.671056 6543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260924 04:41:37.671471 6543 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd3e0098d31455d999a95d5809cefa8; no voters:
I20260924 04:41:37.671813 6543 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:37.671936 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:37.672228 6548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd3e0098d31455d999a95d5809cefa8, State: Running, Role: LEADER
I20260924 04:41:37.672550 6548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260924 04:41:37.673110 6543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:37.674201 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd3e0098d31455d999a95d5809cefa8. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } }
I20260924 04:41:37.674329 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:37.674721 6557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:37.675200 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } }
I20260924 04:41:37.675379 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:37.677474 6557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:37.683401 6557 catalog_manager.cc:1347] Generated new cluster ID: d287a7371e6b4378aa998f2f500332a8
I20260924 04:41:37.683473 6557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:37.707557 6557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:37.708679 6557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:37.716267 6557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Generated new TSK 0
I20260924 04:41:37.717011 6557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:41:37.750929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38757
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:37.867519 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:37.867859 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:37.867944 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:37.872283 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:37.872376 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:37.872469 6569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:37.878430 6569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:37.880211 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:37.881278 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:37.891300 6574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:37.891328 6575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:37.891320 6577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:37.891374 6569 server_base.cc:1034] running on GCE node
I20260924 04:41:37.892014 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:37.892747 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:37.893954 6569 hybrid_clock.cc:648] HybridClock initialized: now 1790224897893915 us; error 51 us; skew 500 ppm
Sep 24 04:41:37 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:37.897042 6569 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:41:37.898007 6569 webserver.cc:460] Webserver started at http://127.0.98.1:38073/ using document root <none> and password file <none>
I20260924 04:41:37.898620 6569 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:37.898684 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:37.898911 6569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:37.901103 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "5faf10a6f2b641e886914ca2c67a591c"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "2e91ea0f37f8c4f355c2a67bb3ade2c8"
server_key_iv: "f7da251d5bffaaa9136a2c2a9cd778c7"
server_key_version: "encryptionkey@0"
I20260924 04:41:37.901684 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5faf10a6f2b641e886914ca2c67a591c"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "2e91ea0f37f8c4f355c2a67bb3ade2c8"
server_key_iv: "f7da251d5bffaaa9136a2c2a9cd778c7"
server_key_version: "encryptionkey@0"
I20260924 04:41:37.905598 6569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:37.908350 6584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:37.909679 6569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:37.909785 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "5faf10a6f2b641e886914ca2c67a591c"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "2e91ea0f37f8c4f355c2a67bb3ade2c8"
server_key_iv: "f7da251d5bffaaa9136a2c2a9cd778c7"
server_key_version: "encryptionkey@0"
I20260924 04:41:37.909933 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:37.940151 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:37.943527 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:37.943748 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:37.944401 6569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:37.945258 6569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:37.945312 6569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:37.945353 6569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:37.945410 6569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:37.954856 6569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34639
I20260924 04:41:37.954878 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34639 every 8 connection(s)
I20260924 04:41:37.955812 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:37.958191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6569
I20260924 04:41:37.958307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:37.958655 392 external_mini_cluster.cc:1428] Setting key 04bbc0251dd2eed97fe88c519987c8e2
Sep 24 04:41:37 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:37.971823 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:37.957857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33527 (local address 127.0.98.62:38757)
0924 04:41:37.958018 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:37.958024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:37.959043 (+ 1019us) server_negotiation.cc:407] Connection header received
0924 04:41:37.960500 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:37.960505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:37.960567 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:37.960661 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:37.961980 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:37.962596 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:37.964032 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:37.964275 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:37.964388 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:37.967005 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:37.967008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:37.967011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:37.967035 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:37.968884 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:37.969736 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:37.969740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:37.969741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:37.969835 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:37.970356 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:37.970360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:37.970362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:37.970580 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:37.970716 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:37.971474 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 04:41:37.971626 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:41:37.972843 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:37.958285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:33527)
0924 04:41:37.958852 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:37.958907 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:37.960258 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:37.960830 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:37.960836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:37.961209 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:37.961811 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:37.961827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:37.962768 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:37.962777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:37.963892 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:37.963902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:37.964432 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:37.964437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:37.964622 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:37.965294 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:37.965311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:37.966816 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:37.969064 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:37.969072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:37.969091 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:37.969568 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:37.969950 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:37.969956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:37.969959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:37.970193 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:37.970704 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:37.970712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:37.971162 (+ 450us) client_negotiation.cc:769] Sending connection context
0924 04:41:37.971431 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 04:41:37.971815 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":159,"threads_started":1}
I20260924 04:41:37.974498 6698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38757
I20260924 04:41:37.974859 6698 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:37.975449 6698 heartbeater.cc:511] Master 127.0.98.62:38757 requested a full tablet report, sending...
I20260924 04:41:37.977252 6507 ts_manager.cc:194] Registered new tserver with Master: 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639)
I20260924 04:41:37.978926 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:41:37.991223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38757
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:38.106832 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:38.107199 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:38.107263 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:38.111382 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:38.111452 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:38.111514 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:38.117226 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:38.118412 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:38.119632 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:38.130156 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.130368 6703 server_base.cc:1034] running on GCE node
W20260924 04:41:38.130139 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.130190 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.131062 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:38.131850 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:38.133059 6703 hybrid_clock.cc:648] HybridClock initialized: now 1790224898133040 us; error 31 us; skew 500 ppm
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:38.136009 6703 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:41:38.136864 6703 webserver.cc:460] Webserver started at http://127.0.98.2:33471/ using document root <none> and password file <none>
I20260924 04:41:38.137429 6703 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:38.137490 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:38.137750 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:38.139950 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d2f8948fdd44b9590a13d124de15253"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "7e4d3240fe07505afa92488673fc7de3"
server_key_iv: "9940a0a44febcd03df2d1397e3addc8b"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.140532 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d2f8948fdd44b9590a13d124de15253"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "7e4d3240fe07505afa92488673fc7de3"
server_key_iv: "9940a0a44febcd03df2d1397e3addc8b"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.144596 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:38.147387 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.148833 6703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:41:38.148945 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "8d2f8948fdd44b9590a13d124de15253"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "7e4d3240fe07505afa92488673fc7de3"
server_key_iv: "9940a0a44febcd03df2d1397e3addc8b"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.149044 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:38.170065 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:38.173766 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:38.173974 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:38.174664 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:38.175721 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:38.175787 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.175861 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:38.175895 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.184594 6703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35289
I20260924 04:41:38.184734 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35289 every 8 connection(s)
I20260924 04:41:38.185647 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:38.188227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6703
I20260924 04:41:38.188336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:38.188654 392 external_mini_cluster.cc:1428] Setting key 5467186ad42d7a70d0b862ac59d657c9
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:41:38.202365 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.187642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47213 (local address 127.0.98.62:38757)
0924 04:41:38.187844 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.187848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.189402 (+ 1554us) server_negotiation.cc:407] Connection header received
0924 04:41:38.190983 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.190985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.191027 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.191095 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.192619 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.193415 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.195431 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.195642 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.195726 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.198600 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.198603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.198605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.198627 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.200204 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.200740 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.200744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.200746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.200826 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.201180 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.201185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.201187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.201372 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.201477 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.202050 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.202166 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:41:38.203603 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.188101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:47213)
0924 04:41:38.189249 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.189296 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.190747 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.191316 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.191329 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.191872 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.192474 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.192488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.193557 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.193564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.195299 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.195313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.195810 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.195816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.196062 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.196714 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.196740 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.198419 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.200340 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.200346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.200362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.200627 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.200929 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.200932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.200934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.201085 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.201475 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.201483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.201784 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.201997 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.202391 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":165,"threads_started":1}
I20260924 04:41:38.204862 6832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38757
I20260924 04:41:38.205121 6832 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:38.205624 6832 heartbeater.cc:511] Master 127.0.98.62:38757 requested a full tablet report, sending...
I20260924 04:41:38.206753 6507 ts_manager.cc:194] Registered new tserver with Master: 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289)
I20260924 04:41:38.207422 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:41:38.220333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38757
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:38.333556 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:38.333870 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:38.333918 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:38.338277 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:38.338336 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:38.338395 6837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:38.343678 6837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:38.344699 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:38.345678 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:38.355464 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.355450 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.355472 6845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.355680 6837 server_base.cc:1034] running on GCE node
I20260924 04:41:38.356199 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:38.356805 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:38.357986 6837 hybrid_clock.cc:648] HybridClock initialized: now 1790224898357943 us; error 52 us; skew 500 ppm
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:38.360847 6837 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:41:38.361789 6837 webserver.cc:460] Webserver started at http://127.0.98.3:36965/ using document root <none> and password file <none>
I20260924 04:41:38.362362 6837 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:38.362450 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:38.362677 6837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:38.364816 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "8fbb6f7977324456ab7632d01606d03e"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "53e875c26f587a1d8c9109519e43958f"
server_key_iv: "e47dfca11f12d8d89347c14daca7c17b"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.365553 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8fbb6f7977324456ab7632d01606d03e"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "53e875c26f587a1d8c9109519e43958f"
server_key_iv: "e47dfca11f12d8d89347c14daca7c17b"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.369658 6837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 04:41:38.372152 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.373383 6837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:38.373553 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "8fbb6f7977324456ab7632d01606d03e"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "53e875c26f587a1d8c9109519e43958f"
server_key_iv: "e47dfca11f12d8d89347c14daca7c17b"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.373675 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:38.387845 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:38.391347 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:38.391553 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:38.392167 6837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:38.393005 6837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:38.393074 6837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.393157 6837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:38.393203 6837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.402450 6837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39157
I20260924 04:41:38.402472 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39157 every 8 connection(s)
I20260924 04:41:38.403434 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:38.407002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6837
I20260924 04:41:38.407194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:38.407507 392 external_mini_cluster.cc:1428] Setting key 79c25fe845725037a6bb237bb469bfa5
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:38.418181 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.405817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44573 (local address 127.0.98.62:38757)
0924 04:41:38.405960 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.405964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.406735 (+ 771us) server_negotiation.cc:407] Connection header received
0924 04:41:38.408487 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.408490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.408542 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.408625 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.409962 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.410502 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.411741 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.411968 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.412084 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.414665 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.414668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.414670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.414695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.416112 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.416635 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.416639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.416642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.416724 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.417066 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.417071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.417073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.417259 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.417375 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.417892 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.418015 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:41:38.419063 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.406063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:44573)
0924 04:41:38.406548 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.406592 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.408263 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.408826 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.408836 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.409245 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.409826 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.409840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.410640 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.410643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.411619 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.411628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.412122 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.412126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.412299 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.412968 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.412984 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.414488 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.416234 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.416241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.416258 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.416527 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.416824 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.416827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.416829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.416969 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.417381 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.417390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.417649 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.417853 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.418187 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260924 04:41:38.420424 6966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38757
I20260924 04:41:38.420702 6966 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:38.421378 6966 heartbeater.cc:511] Master 127.0.98.62:38757 requested a full tablet report, sending...
I20260924 04:41:38.422431 6507 ts_manager.cc:194] Registered new tserver with Master: 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157)
I20260924 04:41:38.422976 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44573
I20260924 04:41:38.433471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:41:38.463056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41423
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38757,127.0.98.61:41423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:38.576951 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:38.577247 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:38.577289 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:38.581522 6971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:38.581590 6971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:38.581610 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:38.581629 6971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:38.581643 6971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:38.587191 6971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38757,127.0.98.61:41423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
W20260924 04:41:38.587450 6971 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38757,127.0.98.61:41423'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:41:38.588371 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:38.589452 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:38.598212 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.598218 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.598276 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.598479 6971 server_base.cc:1034] running on GCE node
I20260924 04:41:38.599171 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:38.600346 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:38.601503 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790224898601487 us; error 34 us; skew 500 ppm
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:38.604432 6971 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:41:38.605449 6971 webserver.cc:460] Webserver started at http://127.0.98.61:36475/ using document root <none> and password file <none>
I20260924 04:41:38.606024 6971 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:38.606117 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:38.606348 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:38.608865 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/instance:
uuid: "6107f15bf1c14dffb36a132e8e566eca"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "3a55938daf9d4d63512d6560cc41a2e4"
server_key_iv: "a28b18257c6b2c9600e5a89cb15f56b4"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.609504 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal/instance:
uuid: "6107f15bf1c14dffb36a132e8e566eca"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "3a55938daf9d4d63512d6560cc41a2e4"
server_key_iv: "a28b18257c6b2c9600e5a89cb15f56b4"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.613435 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:38.616235 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.617580 6971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:38.617770 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
uuid: "6107f15bf1c14dffb36a132e8e566eca"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "3a55938daf9d4d63512d6560cc41a2e4"
server_key_iv: "a28b18257c6b2c9600e5a89cb15f56b4"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.617902 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:38.631908 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:38.635344 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:38.635551 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:38.644865 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41423
I20260924 04:41:38.645141 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41423 every 8 connection(s)
I20260924 04:41:38.645982 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:41:38.649113 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:38.649706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 6971
I20260924 04:41:38.654577 7039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:38.665977 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.651590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.98.61:41423)
0924 04:41:38.651973 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.652000 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.652034 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:41:38.652812 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.652831 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.653185 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.653637 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.654453 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.655609 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.656634 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.657310 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.657481 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.660508 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.660514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.660525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.660554 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.662553 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.663054 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.663060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.663068 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.663263 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.663591 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.663594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.663596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.664135 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.664363 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.664632 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.664945 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":133,"threads_started":1}
I20260924 04:41:38.666823 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.656558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:53172)
0924 04:41:38.656959 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.657008 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.657186 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.657631 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.657638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.658114 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.658253 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.658258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.659040 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.659043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.659874 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.659883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.660508 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.660512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.660622 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.661397 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.661416 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.663163 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.665067 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.665075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.665079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.665305 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.665594 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.665596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.665598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.665687 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.666171 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.666178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.666327 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.666478 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.666698 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
I20260924 04:41:38.666826 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.656600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.0.98.62:38757)
0924 04:41:38.656713 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.656718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.657087 (+ 369us) server_negotiation.cc:407] Connection header received
0924 04:41:38.657342 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.657348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.657410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.657482 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.658376 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.658910 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.660032 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.660265 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.660324 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.663309 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.663315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.663320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.663354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.664945 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.665409 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.665412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.665414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.665501 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.665778 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.665781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.665783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.666035 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.666137 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.666558 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.666678 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:41:38.668349 7039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:41:38.677225 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.669051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:60810)
0924 04:41:38.669237 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.669253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.669343 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.669670 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.669673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.669811 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.669890 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.669895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.670947 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.670953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.671524 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.671531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.671963 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.671966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.672053 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.672491 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.672501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.673944 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.675727 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.675730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.675731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.675913 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.676249 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.676252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.676254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.676340 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.676798 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.676801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.676890 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.676983 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.677105 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 04:41:38.677349 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.669161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.98.61:41423)
0924 04:41:38.669270 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.669276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.669296 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:41:38.669434 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.669439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.669482 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.669563 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.669998 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.670836 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.671650 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.671862 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.671968 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.674087 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.674090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.674092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.674112 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.675624 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.676020 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.676023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.676024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.676138 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.676453 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.676456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.676457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.676634 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.676732 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.677042 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.677171 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":52}
I20260924 04:41:38.680292 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Bootstrap starting.
I20260924 04:41:38.682581 7039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:38.683372 7039 log.cc:824] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:38.685294 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: No bootstrap required, opened a new log
I20260924 04:41:38.688347 7039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } }
I20260924 04:41:38.688609 7039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:38.688695 7039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6107f15bf1c14dffb36a132e8e566eca, State: Initialized, Role: FOLLOWER
I20260924 04:41:38.689163 7039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } }
I20260924 04:41:38.689651 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } }
I20260924 04:41:38.689765 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:38.690364 7039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:41:38.695293 7056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:41:38.695379 7056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:41:38.697427 6971 master_runner.cc:428] Detected that this master 6107f15bf1c14dffb36a132e8e566eca is joining an existing cluster
I20260924 04:41:38.697492 6971 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41423
I20260924 04:41:38.708732 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.699998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.98.61:41423)
0924 04:41:38.700103 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.700107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.700747 (+ 640us) server_negotiation.cc:407] Connection header received
0924 04:41:38.700890 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.700893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.700933 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.701014 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.701487 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.702240 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.703277 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.703550 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.703666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.705066 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.705070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.705073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.705099 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.707012 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707492 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.707498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.707500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.707587 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707860 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.707863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.707864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.708045 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.708154 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.708408 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.708564 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":29,"spinlock_wait_cycles":1920}
I20260924 04:41:38.708743 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.699840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:60820)
0924 04:41:38.700677 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.700692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.700793 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.701026 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.701029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.701231 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.701370 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.701377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.702355 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.702358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.703105 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.703164 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.703713 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.703716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.703895 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.704398 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.704414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.704924 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.707133 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.707157 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.707359 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.707676 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.707684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.707765 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.708153 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.708156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.708257 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.708366 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.708514 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":85,"spinlock_wait_cycles":2944,"thread_start_us":86,"threads_started":1}
I20260924 04:41:38.709093 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.700137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.0.98.62:38757)
0924 04:41:38.700279 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.700284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.700404 (+ 120us) server_negotiation.cc:407] Connection header received
0924 04:41:38.700567 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.700571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.700646 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.700736 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.701311 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.701899 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.702973 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.703384 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.703496 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.705415 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.705418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.705421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.705450 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.707292 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707790 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.707793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.707795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.707877 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.708184 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.708187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.708189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.708394 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.708462 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.708815 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.708943 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:38.709589 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.699840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:53186)
0924 04:41:38.700301 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.700324 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.700450 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.700779 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.700782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.701029 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.701174 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.701182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.702004 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.702010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.702845 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.702854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.703528 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.703531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.703623 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.704266 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.704300 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.704924 (+ 624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.707416 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.707421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.707675 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.707996 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.707998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.708000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.708083 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.708531 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.708534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.708637 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.709280 (+ 643us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.709416 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":23,"thread_start_us":112,"threads_started":1}
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:41:38.724399 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.714049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.0.98.61:41423)
0924 04:41:38.714174 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.714179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.714526 (+ 347us) server_negotiation.cc:407] Connection header received
0924 04:41:38.714723 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.714726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.714783 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.714867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.715539 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.716321 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.717270 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.717608 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.717715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.720328 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.720332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.720335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.720361 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.722077 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.722679 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.722682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.722684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.722770 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.723155 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.723159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.723160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.723417 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.723510 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.724098 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.724224 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:41:38.725072 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.714018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:60824)
0924 04:41:38.714399 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.714449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.714578 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.714890 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.714894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.715306 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.715430 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.715436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.716445 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.716448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.717142 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.717151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.717719 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.717721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.717814 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.718371 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.718383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.720187 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.722212 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.722223 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.722233 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.722578 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.722936 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.722939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.722941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.723033 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.723565 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.723573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.723722 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.723874 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.724904 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":120,"threads_started":1}
I20260924 04:41:38.726876 6507 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41423: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41423
}
attrs {
promote: true
}
}
dest_uuid: "6bd3e0098d31455d999a95d5809cefa8"
cas_config_opid_index: -1
I20260924 04:41:38.727581 6507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:41:38.729522 7068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } }
I20260924 04:41:38.730110 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:38.731349 7068 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41423
I20260924 04:41:38.731451 7010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Refusing update from remote peer 6bd3e0098d31455d999a95d5809cefa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:41:38.731554 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } }
I20260924 04:41:38.731659 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260924 04:41:38.732448 6971 master.cc:537] Master@127.0.98.61:41423 shutting down...
I20260924 04:41:38.732525 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:41:38.734575 6493 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 04:41:38.735751 6493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 -> Peer 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61:41423): Couldn't send request to peer 6107f15bf1c14dffb36a132e8e566eca. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:41:38.737814 6971 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:38.738116 6971 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:38.738229 6971 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: stopping tablet replica
I20260924 04:41:38.743584 6971 master.cc:552] Master@127.0.98.61:41423 shutdown complete.
I20260924 04:41:38.743654 6971 master_runner.cc:305] Clearing existing system tablet
I20260924 04:41:38.743897 6971 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:38.743988 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:38.745378 6971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:41:38.746389 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.746661 6971 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:38.746788 6971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:38.746822 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
uuid: "6107f15bf1c14dffb36a132e8e566eca"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "3a55938daf9d4d63512d6560cc41a2e4"
server_key_iv: "a28b18257c6b2c9600e5a89cb15f56b4"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.749341 6971 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:38.752432 6971 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:41:38.752526 6971 log.cc:1196] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:41:38.753480 6971 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Deleting consensus metadata
I20260924 04:41:38.753695 6971 master_runner.cc:315] Copying system tablet from 127.0.98.62:38757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:41:38.754916 6971 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38757
I20260924 04:41:38.755306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39651
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:41:38.766620 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.756953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.98.62:38757)
0924 04:41:38.757081 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.757087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.757356 (+ 269us) server_negotiation.cc:407] Connection header received
0924 04:41:38.757537 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.757540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.757608 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.757712 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.758336 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.759179 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.760383 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.760745 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.760825 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.762868 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.762872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.762876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.762905 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.764737 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.765237 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.765241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.765243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.765324 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.765653 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.765657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.765659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.765881 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.765956 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.766322 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.766448 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:41:38.767238 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.756850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:53190)
0924 04:41:38.757261 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.757280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.757417 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.757835 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.757839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.758059 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.758180 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.758187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.759300 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.759304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.760233 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.760244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.760996 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.761000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.761095 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.762181 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.762196 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.762694 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.764870 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.764874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.764876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.765126 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.765433 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.765437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.765439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.765522 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.766027 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.766029 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.766136 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.766859 (+ 723us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.767014 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":99,"threads_started":1}
I20260924 04:41:38.767961 6527 tablet_copy_service.cc:140] P 6bd3e0098d31455d999a95d5809cefa8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6107f15bf1c14dffb36a132e8e566eca ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53190)
I20260924 04:41:38.768158 6527 tablet_copy_service.cc:161] P 6bd3e0098d31455d999a95d5809cefa8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6107f15bf1c14dffb36a132e8e566eca at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53190: session id = 6107f15bf1c14dffb36a132e8e566eca-00000000000000000000000000000000
I20260924 04:41:38.770059 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:41:38.771945 6971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:41:38.774379 6971 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: tablet copy: Starting download of 0 data blocks...
I20260924 04:41:38.774578 6971 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: tablet copy: Starting download of 1 WAL segments...
I20260924 04:41:38.776497 6971 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:41:38.778831 6527 tablet_copy_service.cc:342] P 6bd3e0098d31455d999a95d5809cefa8: Request end of tablet copy session 6107f15bf1c14dffb36a132e8e566eca-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53190
I20260924 04:41:38.779027 6527 tablet_copy_service.cc:433] P 6bd3e0098d31455d999a95d5809cefa8: ending tablet copy session 6107f15bf1c14dffb36a132e8e566eca-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6107f15bf1c14dffb36a132e8e566eca
I20260924 04:41:38.787086 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:38.788936 7081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.789021 7084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.789371 6971 server_base.cc:1034] running on GCE node
W20260924 04:41:38.789578 7082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.789855 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:38.790180 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:38.791304 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790224898791294 us; error 30 us; skew 500 ppm
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:38.792972 6971 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:41:38.793426 6971 webserver.cc:460] Webserver started at http://127.0.98.61:43785/ using document root <none> and password file <none>
I20260924 04:41:38.793609 6971 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:38.793651 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:38.795034 6971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:38.796092 7091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.796394 6971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:38.796468 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
uuid: "6107f15bf1c14dffb36a132e8e566eca"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "3a55938daf9d4d63512d6560cc41a2e4"
server_key_iv: "a28b18257c6b2c9600e5a89cb15f56b4"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.796553 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:38.814877 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:38.817023 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:38.821717 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41423
I20260924 04:41:38.822034 7143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41423 every 8 connection(s)
I20260924 04:41:38.822319 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:41:38.824234 7144 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:41:38.825372 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Bootstrap starting.
I20260924 04:41:38.836424 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:41:38.836807 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: Bootstrap complete.
I20260924 04:41:38.837442 7144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } }
I20260924 04:41:38.837590 7144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6107f15bf1c14dffb36a132e8e566eca, State: Initialized, Role: LEARNER
I20260924 04:41:38.837733 7144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } }
I20260924 04:41:38.838129 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } }
I20260924 04:41:38.838241 7144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:38.838321 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: LEARNER
I20260924 04:41:38.841586 7158 catalog_manager.cc:1259] Loaded cluster ID: d287a7371e6b4378aa998f2f500332a8
I20260924 04:41:38.841727 7158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: loading cluster ID for follower catalog manager: success
I20260924 04:41:38.843954 7158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: acquiring CA information for follower catalog manager: success
I20260924 04:41:38.845407 7158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:41:38.893311 7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:38.893597 7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:38.893648 7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:38.898120 7079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:38.898200 7079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:38.898221 7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:38.898238 7079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:38.898286 7079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:38.903959 7079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:38.905203 7079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:38.906270 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:38.915357 7164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.915462 7163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:38.915474 7166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:38.915643 7079 server_base.cc:1034] running on GCE node
I20260924 04:41:38.916250 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:38.917399 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:38.918591 7079 hybrid_clock.cc:648] HybridClock initialized: now 1790224898918547 us; error 58 us; skew 500 ppm
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:38.921465 7079 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:41:38.922298 7079 webserver.cc:460] Webserver started at http://127.0.98.60:36653/ using document root <none> and password file <none>
I20260924 04:41:38.922830 7079 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:38.922889 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:38.923056 7079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:38.925282 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/instance:
uuid: "bf4120f6949c40b8a468142e68d644b4"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "e2fa94556823d0f4ef60a2743733b5cc"
server_key_iv: "87443d993ee147553125a9e21124d52f"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.925866 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal/instance:
uuid: "bf4120f6949c40b8a468142e68d644b4"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "e2fa94556823d0f4ef60a2743733b5cc"
server_key_iv: "87443d993ee147553125a9e21124d52f"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.929965 7079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:38.932804 7173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:38.934027 7079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:38.934195 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
uuid: "bf4120f6949c40b8a468142e68d644b4"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "e2fa94556823d0f4ef60a2743733b5cc"
server_key_iv: "87443d993ee147553125a9e21124d52f"
server_key_version: "encryptionkey@0"
I20260924 04:41:38.934314 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:38.958177 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:38.961632 7079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:38.961848 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:38.969453 7079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39651
I20260924 04:41:38.969518 7225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39651 every 8 connection(s)
I20260924 04:41:38.970487 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:41:38.973683 7226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:38.974610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 7079
I20260924 04:41:38.978489 7226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:41:38.981688 6698 heartbeater.cc:503] Master 127.0.98.62:38757 was elected leader, sending a full tablet report...
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:38.990543 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.980646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.98.62:38757)
0924 04:41:38.980796 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.980801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.981044 (+ 243us) server_negotiation.cc:407] Connection header received
0924 04:41:38.981284 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.981287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.981335 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.981405 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.982407 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.983061 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.984309 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.984547 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.984662 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.987269 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.987272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.987275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.987305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.988609 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.989123 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.989127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.989128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.989202 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.989528 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.989532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.989534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.989722 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.989834 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.990218 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.990345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:38.991528 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.976368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.0.98.60:39651)
0924 04:41:38.976886 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.976901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.976937 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:41:38.977727 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.977757 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.978199 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.978564 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.979471 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.980977 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.982411 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.983192 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.983361 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.986156 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.986162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.986174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.986196 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:38.988383 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.988901 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.988915 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.988917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.989032 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:38.989347 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:38.989350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:38.989352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:38.989777 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:38.990025 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:38.990208 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:41:38.990530 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":367,"thread_start_us":145,"threads_started":1}
I20260924 04:41:38.991526 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.980434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:53200)
0924 04:41:38.980907 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.980950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.981114 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.981492 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.981499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.982061 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.982238 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.982246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.983231 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.983237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.984181 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.984190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.984705 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.984709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.984820 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.985396 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.985417 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.987100 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:38.988741 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.988752 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.988763 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.988994 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.989311 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:38.989314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:38.989316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:38.989434 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:38.989830 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:38.989837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:38.989991 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 04:41:38.990177 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:41:38.990727 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":201,"thread_start_us":151,"threads_started":1}
I20260924 04:41:38.993155 7226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:41:38 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:41:39.002985 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.993695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:60840)
0924 04:41:38.993839 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:38.993855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:38.993927 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:38.994320 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:38.994325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:38.994505 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:38.995025 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.995033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.996115 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.996118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.996762 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:38.996772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.997412 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.997415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:38.997515 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.998031 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:38.998050 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:38.999454 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.001379 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.001382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.001383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.001596 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.001939 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.001942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.001943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.002036 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.002505 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.002507 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.002612 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.002708 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.002833 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:41:39.003087 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:38.993774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.98.61:41423)
0924 04:41:38.994093 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:38.994097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:38.994109 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:38.994151 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:38.994153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:38.994193 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:38.994272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:38.995227 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.995956 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.996924 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:38.997243 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:38.997388 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:38.999601 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:38.999605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:38.999607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:38.999632 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.001259 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.001691 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.001693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.001695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.001831 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.002180 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.002183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.002184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.002391 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.002473 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.002818 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.002952 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260924 04:41:39.004061 7226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:41:39.012355 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.004703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.98.60:39651)
0924 04:41:39.004831 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.004836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.004916 (+ 80us) server_negotiation.cc:407] Connection header received
0924 04:41:39.005055 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.005058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.005097 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.005177 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.005558 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.006065 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.006804 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.007031 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.007157 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.009176 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.009179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.009181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.009200 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.010700 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.011154 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.011158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.011159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.011253 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.011525 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.011528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.011529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.011706 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.011806 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.012038 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.012218 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":50}
I20260924 04:41:39.012355 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.004625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:55234)
0924 04:41:39.004831 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.004847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.004959 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.005185 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.005187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.005386 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.005451 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.005456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.006164 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.006170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.006685 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.006691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.007164 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.007167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.007258 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.007714 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.007726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.009024 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.010827 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.010833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.010836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.011023 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.011358 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.011361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.011365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.011434 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.011795 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.011797 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.011886 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.011996 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.012148 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":4}
I20260924 04:41:39.016394 7226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Bootstrap starting.
I20260924 04:41:39.019061 7226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:39.019990 7226 log.cc:824] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:39.022409 7226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: No bootstrap required, opened a new log
I20260924 04:41:39.026019 7226 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } }
I20260924 04:41:39.026279 7226 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:39.026324 7226 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4120f6949c40b8a468142e68d644b4, State: Initialized, Role: FOLLOWER
I20260924 04:41:39.026873 7226 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } }
I20260924 04:41:39.027494 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } } }
I20260924 04:41:39.027642 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:39.028391 7226 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:41:39.033636 7243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:41:39.034193 7243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:41:39.038041 7079 master_runner.cc:428] Detected that this master bf4120f6949c40b8a468142e68d644b4 is joining an existing cluster
I20260924 04:41:39.038110 7079 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39651
I20260924 04:41:39.045428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6569
I20260924 04:41:39.055989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34639
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38757
--webserver_port=38073
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:41:39.073925 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.041351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.0.98.62:38757)
0924 04:41:39.041499 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.041503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.041515 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:39.041559 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.041561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.041600 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.041732 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.042275 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.043146 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.044308 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.044697 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.044785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.048059 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.048063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.048066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.048099 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.049936 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.072086 (+ 22150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.072094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.072098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.072228 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.072601 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.072605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.072608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.073003 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.073093 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.073556 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.073739 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:41:39.074589 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.040753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:53212)
0924 04:41:39.041118 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.041135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.041263 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.041770 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.041773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.041986 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.042123 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.042133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.043282 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.043285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.044180 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.044187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.044946 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.044950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.046595 (+ 1645us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.047183 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.047200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.047842 (+ 642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.050086 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.050092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.050097 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.071864 (+ 21767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.072368 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.072375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.072379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.072479 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.073177 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.073183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.073332 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.074193 (+ 861us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.074359 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":1535,"thread_start_us":149,"threads_started":1}
I20260924 04:41:39.076715 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.041719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.0.98.61:41423)
0924 04:41:39.041851 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.041856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.042398 (+ 542us) server_negotiation.cc:407] Connection header received
0924 04:41:39.042599 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.042602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.042658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.042717 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.043493 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.044385 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.052499 (+ 8114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.052769 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.052903 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.072172 (+ 19269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.072179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.072182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.072215 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.074265 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.074794 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.074798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.074800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.074891 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.075785 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.075789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.075790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.076008 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.076093 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.076440 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.076567 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:39.078265 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.041959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:60852)
0924 04:41:39.042304 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.042319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.042472 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.042879 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.042882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.043173 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.043342 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.043351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.044554 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.044558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.052374 (+ 7816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.052383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.052900 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.052903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.053017 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.053532 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.053546 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.071964 (+ 18418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.074427 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.074430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.074432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.074665 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.075018 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.075021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.075022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.075108 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.076159 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.076162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.076257 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.076973 (+ 716us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.077422 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":23,"thread_start_us":109,"threads_started":1}
I20260924 04:41:39.079382 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.041187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.0.98.60:39651)
0924 04:41:39.041291 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.041297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.042350 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 04:41:39.042478 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.042480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.042512 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.042554 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.043076 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.043924 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.046482 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.047213 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.047324 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.049195 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.049201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.049205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.049239 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.050710 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.077719 (+ 27009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.077723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.077724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.077807 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.078412 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.078415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.078417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.078643 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.078720 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.079064 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.079232 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":48}
I20260924 04:41:39.079912 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.040835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:55244)
0924 04:41:39.042276 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.042291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.042398 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.042659 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.042662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.042868 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.042970 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.042977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.044315 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.044318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.046354 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.046363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.047426 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.047429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.047514 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.048038 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.048052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.048762 (+ 710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.050811 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.050813 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.050814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.077295 (+ 26481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.078023 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.078025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.078027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.078121 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.078781 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.078783 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.078885 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.079602 (+ 717us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.079750 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":109,"threads_started":1}
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224897, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:41:39.087406 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.079023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:55258)
0924 04:41:39.079178 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.079195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.079279 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.079886 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.079889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.080094 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.080258 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.080266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.081135 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.081138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.081695 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.081703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.082164 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.082167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.082265 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.082782 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.082794 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.084134 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.085888 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.085891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.085893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.086095 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.086377 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.086381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.086383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.086468 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.086840 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.086843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.086945 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.087047 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.087225 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:41:39.087406 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.079315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.0.98.60:39651)
0924 04:41:39.079541 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.079545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.079559 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:39.079629 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.079632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.079683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.079761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.080381 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.081009 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.081817 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.082033 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.082146 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.084279 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.084287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.084289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.084311 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.085726 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.086195 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.086200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.086202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.086283 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.086557 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.086561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.086563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.086743 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.086849 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.087108 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.087262 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260924 04:41:39.088760 6507 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39651: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bf4120f6949c40b8a468142e68d644b4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39651
}
attrs {
promote: true
}
}
dest_uuid: "6bd3e0098d31455d999a95d5809cefa8"
cas_config_opid_index: 5
I20260924 04:41:39.089337 6507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } }
I20260924 04:41:39.090649 7067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bf4120f6949c40b8a468142e68d644b4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.091826 7200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:39.092660 7067 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39651
I20260924 04:41:39.092833 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.092931 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.093364 7200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Refusing update from remote peer 6bd3e0098d31455d999a95d5809cefa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:41:39.093364 7079 master.cc:537] Master@127.0.98.60:39651 shutting down...
I20260924 04:41:39.094484 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:39.095726 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.090874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:60866)
0924 04:41:39.091049 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.091065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.091170 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.091575 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.091579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.091597 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:39.091684 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.091715 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.092799 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.092803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.094105 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.094116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.095267 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.095271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.095374 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.095418 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.095500 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.095573 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 04:41:39.096185 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.090989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.0.98.61:41423)
0924 04:41:39.091113 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.091153 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.091169 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:41:39.091264 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.091268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.091333 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.091446 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:39.091836 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.092649 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.094255 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.094913 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.095018 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.095205 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.095919 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.096005 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:41:39.096585 7118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 LEARNER]: Refusing update from remote peer 6bd3e0098d31455d999a95d5809cefa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260924 04:41:39.096685 6492 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 -> Peer bf4120f6949c40b8a468142e68d644b4 (127.0.98.60:39651): Couldn't send request to peer bf4120f6949c40b8a468142e68d644b4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:41:39.097711 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:41:39.098839 7079 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:39.099162 7079 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:39.099275 7079 tablet_replica.cc:330] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: stopping tablet replica
I20260924 04:41:39.099860 7118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bf4120f6949c40b8a468142e68d644b4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.101536 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.101661 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: LEARNER
I20260924 04:41:39.101521 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd3e0098d31455d999a95d5809cefa8. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: true } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.101903 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: LEARNER
I20260924 04:41:39.105376 7254 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: attempting to promote NON_VOTER 6107f15bf1c14dffb36a132e8e566eca to VOTER
I20260924 04:41:39.105829 7254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } }
I20260924 04:41:39.107038 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.106762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:55270)
0924 04:41:39.106881 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.106954 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:41:39.107466 7118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 LEARNER]: Refusing update from remote peer 6bd3e0098d31455d999a95d5809cefa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:41:39.107918 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.107716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:55286)
0924 04:41:39.107805 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.107853 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 04:41:39.108016 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 04:41:39.108317 6492 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 -> Peer bf4120f6949c40b8a468142e68d644b4 (127.0.98.60:39651): Couldn't send request to peer bf4120f6949c40b8a468142e68d644b4. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:41:39.109543 7067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.110773 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.110893 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.111019 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.111148 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.111726 7118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.112996 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.113123 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:39.116392 7079 master.cc:552] Master@127.0.98.60:39651 shutdown complete.
I20260924 04:41:39.116459 7079 master_runner.cc:305] Clearing existing system tablet
I20260924 04:41:39.116683 7079 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:39.116760 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:39.117851 7079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:41:39.118692 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.118934 7079 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:39.119091 7079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:39.119161 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
uuid: "bf4120f6949c40b8a468142e68d644b4"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "e2fa94556823d0f4ef60a2743733b5cc"
server_key_iv: "87443d993ee147553125a9e21124d52f"
server_key_version: "encryptionkey@0"
I20260924 04:41:39.120836 7079 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:39.122794 7079 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:41:39.122901 7079 log.cc:1196] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:41:39.123227 7079 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Deleting consensus metadata
I20260924 04:41:39.123437 7079 master_runner.cc:315] Copying system tablet from 127.0.98.62:38757
I20260924 04:41:39.124646 7079 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38757
I20260924 04:41:39.133062 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.125292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.0.98.62:38757)
0924 04:41:39.125423 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.125428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.125632 (+ 204us) server_negotiation.cc:407] Connection header received
0924 04:41:39.125798 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.125801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.125860 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.125952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.126417 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.126950 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.127922 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.128152 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.128264 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.129700 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.129703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.129706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.129732 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.131345 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.131844 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.131850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.131851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.131943 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.132216 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.132221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.132223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.132406 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.132512 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.132784 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.132899 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:41:39.133074 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.125239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:53226)
0924 04:41:39.125536 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.125548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.125684 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.125982 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.125984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.126190 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.126296 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.126302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.127061 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.127068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.127790 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.127800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.128275 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.128291 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.128443 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.129068 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.129082 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.129577 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.131470 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.131473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.131475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.131741 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.132036 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.132040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.132042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.132129 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.132501 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.132503 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.132608 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.132712 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.132858 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
I20260924 04:41:39.134030 6527 tablet_copy_service.cc:140] P 6bd3e0098d31455d999a95d5809cefa8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bf4120f6949c40b8a468142e68d644b4 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53226)
I20260924 04:41:39.134161 6527 tablet_copy_service.cc:161] P 6bd3e0098d31455d999a95d5809cefa8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bf4120f6949c40b8a468142e68d644b4 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53226: session id = bf4120f6949c40b8a468142e68d644b4-00000000000000000000000000000000
I20260924 04:41:39.135008 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:41:39.136215 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:41:39.138370 7079 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: tablet copy: Starting download of 0 data blocks...
I20260924 04:41:39.138641 7079 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: tablet copy: Starting download of 1 WAL segments...
I20260924 04:41:39.140187 7079 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:41:39.141821 6527 tablet_copy_service.cc:342] P 6bd3e0098d31455d999a95d5809cefa8: Request end of tablet copy session bf4120f6949c40b8a468142e68d644b4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53226
I20260924 04:41:39.141923 6527 tablet_copy_service.cc:433] P 6bd3e0098d31455d999a95d5809cefa8: ending tablet copy session bf4120f6949c40b8a468142e68d644b4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bf4120f6949c40b8a468142e68d644b4
I20260924 04:41:39.149369 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:39.151113 7265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:39.151172 7268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.151662 7079 server_base.cc:1034] running on GCE node
W20260924 04:41:39.151662 7266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.152118 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:39.152490 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:39.153728 7079 hybrid_clock.cc:648] HybridClock initialized: now 1790224899153721 us; error 31 us; skew 500 ppm
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:39.155400 7079 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:41:39.155907 7079 webserver.cc:460] Webserver started at http://127.0.98.60:36743/ using document root <none> and password file <none>
I20260924 04:41:39.156112 7079 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:39.156168 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:39.157308 7079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:39.158268 7275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.158552 7079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:39.158659 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
uuid: "bf4120f6949c40b8a468142e68d644b4"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "e2fa94556823d0f4ef60a2743733b5cc"
server_key_iv: "87443d993ee147553125a9e21124d52f"
server_key_version: "encryptionkey@0"
I20260924 04:41:39.158780 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:39.168859 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:39.170912 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:39.176586 7079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39651
I20260924 04:41:39.177207 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:41:39.177591 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39651 every 8 connection(s)
I20260924 04:41:39.178851 7328 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:41:39.179836 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Bootstrap starting.
I20260924 04:41:39.189828 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:41:39.190238 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Bootstrap complete.
I20260924 04:41:39.190828 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } }
I20260924 04:41:39.190968 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf4120f6949c40b8a468142e68d644b4, State: Initialized, Role: LEARNER
I20260924 04:41:39.191216 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } }
I20260924 04:41:39.191601 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.191742 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:41:39.191864 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:39.194629 7342 catalog_manager.cc:1259] Loaded cluster ID: d287a7371e6b4378aa998f2f500332a8
I20260924 04:41:39.194881 7342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: loading cluster ID for follower catalog manager: success
I20260924 04:41:39.196835 7342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: acquiring CA information for follower catalog manager: success
I20260924 04:41:39.198082 7342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:41:39.209766 6832 heartbeater.cc:503] Master 127.0.98.62:38757 was elected leader, sending a full tablet report...
W20260924 04:41:39.231308 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:39.231711 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:39.231782 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:39.238229 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:39.238317 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:39.238408 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:39.247478 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38073
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:39.248842 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:39.250128 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:39.261977 7348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:39.261982 7350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.262157 7252 server_base.cc:1034] running on GCE node
W20260924 04:41:39.261982 7347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.262732 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:39.263505 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:39.264706 7252 hybrid_clock.cc:648] HybridClock initialized: now 1790224899264671 us; error 44 us; skew 500 ppm
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:39.268134 7252 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:41:39.269076 7252 webserver.cc:460] Webserver started at http://127.0.98.1:38073/ using document root <none> and password file <none>
I20260924 04:41:39.269662 7252 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:39.269785 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:39.274039 7252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:39.276281 7357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.277814 7252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:39.277949 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "5faf10a6f2b641e886914ca2c67a591c"
format_stamp: "Formatted at 2026-09-24 04:41:37 on dist-test-slave-npg2"
server_key: "2e91ea0f37f8c4f355c2a67bb3ade2c8"
server_key_iv: "f7da251d5bffaaa9136a2c2a9cd778c7"
server_key_version: "encryptionkey@0"
I20260924 04:41:39.278512 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:39.299093 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:39.302227 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:39.302429 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:39.303041 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:39.303956 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:39.304001 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.304071 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:39.304103 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.314129 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34639
I20260924 04:41:39.314177 7470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34639 every 8 connection(s)
I20260924 04:41:39.315274 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:39.324733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 7252
I20260924 04:41:39.324971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6703
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:41:39.333052 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.317891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59445 (local address 127.0.98.62:38757)
0924 04:41:39.318035 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.318041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.319428 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 04:41:39.319871 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.319874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.319934 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.319988 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.321454 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.321990 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.323684 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.324045 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.324151 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.328022 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.328026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.328030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.328064 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.330981 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331547 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.331554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.331559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.331650 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331959 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.331963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.331964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.332188 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.332284 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.332746 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.332888 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:41:39.333182 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.317906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42841 (local address 127.0.98.61:41423)
0924 04:41:39.318042 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.318047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.318712 (+ 665us) server_negotiation.cc:407] Connection header received
0924 04:41:39.319844 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.319847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.319903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.319983 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.321470 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.322283 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.323606 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.323936 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.324025 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.329758 (+ 5733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.329764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.329768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.329801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.331509 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331962 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.331965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.331966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.332038 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.332328 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.332331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.332332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.332528 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.332609 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.332934 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.333038 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:41:39.334198 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.317873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:59445)
0924 04:41:39.319336 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.319348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.319671 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.320162 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.320170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.320719 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.321304 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.321320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.322131 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.322134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.323538 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.323547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.324284 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.324287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.324443 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.325624 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.325642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.327798 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.331147 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.331153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.331423 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.331762 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.331771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.331853 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.332335 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.332338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.332463 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.333271 (+ 808us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.333478 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":137,"spinlock_wait_cycles":16384,"thread_start_us":301,"threads_started":1}
I20260924 04:41:39.334201 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.317512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.98.1:42841)
0924 04:41:39.318522 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.318604 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.319637 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.320160 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.320169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.320741 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.321317 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.321324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.322394 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.322398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.323460 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.323470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.324145 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.324148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.324273 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.325010 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.325069 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.329573 (+ 4504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.331621 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.331626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.331858 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.332142 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.332145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.332148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.332225 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.332667 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.332673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.332762 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.333488 (+ 726us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.333613 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":122,"spinlock_wait_cycles":9984,"thread_start_us":1038,"threads_started":1}
I20260924 04:41:39.334645 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.317101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39585 (local address 127.0.98.60:39651)
0924 04:41:39.317417 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.317431 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.319110 (+ 1679us) server_negotiation.cc:407] Connection header received
0924 04:41:39.319962 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.319968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.320022 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.320107 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.321471 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.322513 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.323532 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.323885 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.324006 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.328360 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.328364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.328367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.328397 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.330376 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331050 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.331056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.331061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.331190 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331670 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.331676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.331681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.331958 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.332051 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.334325 (+ 2274us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.334461 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260924 04:41:39.334672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35289
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38757
--webserver_port=33471
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:41:39.334205 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.317512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.98.1:39585)
0924 04:41:39.319001 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.319015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.319653 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.320160 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.320169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.320740 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.321314 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.321324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.322619 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.322625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.323411 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.323420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.324036 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.324039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.324283 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.326077 (+ 1794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.326091 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.328192 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.330511 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.330521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.330543 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.330929 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.331330 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.331336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.331341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.331552 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.332106 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.332116 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.332466 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.332670 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.333082 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":258,"spinlock_wait_cycles":69248,"thread_start_us":523,"threads_started":1}
I20260924 04:41:39.335541 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38757
I20260924 04:41:39.335783 7472 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.336390 7472 heartbeater.cc:511] Master 127.0.98.62:38757 requested a full tablet report, sending...
I20260924 04:41:39.336725 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41423
I20260924 04:41:39.336831 7473 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.337019 7473 heartbeater.cc:511] Master 127.0.98.61:41423 requested a full tablet report, sending...
I20260924 04:41:39.337373 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39651
I20260924 04:41:39.337446 7471 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.337592 7471 heartbeater.cc:511] Master 127.0.98.60:39651 requested a full tablet report, sending...
I20260924 04:41:39.337677 6507 ts_manager.cc:194] Re-registered known tserver with Master: 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639)
I20260924 04:41:39.338454 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59445
I20260924 04:41:39.338496 7108 ts_manager.cc:194] Registered new tserver with Master: 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639)
I20260924 04:41:39.338675 7283 ts_manager.cc:194] Registered new tserver with Master: 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639)
I20260924 04:41:39.425423 6966 heartbeater.cc:503] Master 127.0.98.62:38757 was elected leader, sending a full tablet report...
W20260924 04:41:39.454959 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:39.455317 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:39.455402 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:39.459918 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:39.460009 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:39.460100 7479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:39.465713 7479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33471
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:39.466863 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:39.468014 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:39.478209 7487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:39.478209 7485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:39.478230 7484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.478689 7479 server_base.cc:1034] running on GCE node
I20260924 04:41:39.479063 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:39.479770 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:39.480965 7479 hybrid_clock.cc:648] HybridClock initialized: now 1790224899480928 us; error 47 us; skew 500 ppm
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:39.484195 7479 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:41:39.485117 7479 webserver.cc:460] Webserver started at http://127.0.98.2:33471/ using document root <none> and password file <none>
I20260924 04:41:39.485921 7479 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:39.486039 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:39.489316 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.485551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:55300)
0924 04:41:39.485682 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.485698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.485794 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.486086 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.486091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.486112 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:39.486218 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.486223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.487329 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.487332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.488070 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.488079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.489003 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.489005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.489066 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.489086 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.489130 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.489188 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:41:39.489395 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.485652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.0.98.60:39651)
0924 04:41:39.485795 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.485800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.485817 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:41:39.485877 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.485881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.485945 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.486052 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:39.486373 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.487218 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.488203 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.488877 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.488996 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.489165 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.489204 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.489279 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:41:39.489943 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 04:41:39.490408 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd3e0098d31455d999a95d5809cefa8. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.490587 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:41:39.491211 7479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:41:39.493451 7494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.494735 7479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:39.494910 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "8d2f8948fdd44b9590a13d124de15253"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "7e4d3240fe07505afa92488673fc7de3"
server_key_iv: "9940a0a44febcd03df2d1397e3addc8b"
server_key_version: "encryptionkey@0"
I20260924 04:41:39.495500 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:39.523298 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:39.526881 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:39.527169 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:39.527879 7479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:39.528992 7479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:39.529060 7479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.529129 7479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:39.529160 7479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.539002 7479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35289
I20260924 04:41:39.539052 7607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35289 every 8 connection(s)
I20260924 04:41:39.540373 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:39.544041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 7479
I20260924 04:41:39.544224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6837
I20260924 04:41:39.551867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39157
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38757
--webserver_port=36965
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:41:39.562143 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.542541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50757 (local address 127.0.98.62:38757)
0924 04:41:39.543875 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.543881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.543901 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:41:39.546503 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.546509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.546575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.546633 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.548943 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.549800 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.551956 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.552341 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.552437 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.556365 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.556370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.556375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.556409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.559154 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560103 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.560108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.560109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.560209 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560690 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.560694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.560696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.560919 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.561003 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.561845 (+ 842us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.561976 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224}
I20260924 04:41:39.562143 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.542881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38507 (local address 127.0.98.60:39651)
0924 04:41:39.543015 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.543019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.543951 (+ 932us) server_negotiation.cc:407] Connection header received
0924 04:41:39.546508 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.546512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.546570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.546625 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.548922 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.549767 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.551932 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.552278 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.552357 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.557151 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.557157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.557162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.557209 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.559039 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560124 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.560127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.560129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.560218 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560682 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.560685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.560687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.560919 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.561003 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.561845 (+ 842us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.561976 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:39.562265 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.542881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53717 (local address 127.0.98.61:41423)
0924 04:41:39.543015 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.543019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.544234 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 04:41:39.546508 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.546512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.546570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.546625 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.548935 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.549759 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.551931 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.552286 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.552357 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.557591 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.557595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.557598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.557631 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.559405 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560166 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.560172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.560176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.560288 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560697 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.560702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.560704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.560965 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.561061 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.561866 (+ 805us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.562023 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:41:39.564065 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.542763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.98.2:53717)
0924 04:41:39.544143 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.544155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.546256 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.547055 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.547066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.547868 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.548730 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.548752 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.549955 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.549959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.551778 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.551791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.552713 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.552720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.553024 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.553899 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.553916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.557453 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.559531 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.559538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.559542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.560024 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.560417 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.560425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.560575 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.561125 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.561129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.561540 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.562419 (+ 879us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.562932 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":1014,"spinlock_wait_cycles":34560,"thread_start_us":182,"threads_started":1}
I20260924 04:41:39.564061 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.542763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.98.2:38507)
0924 04:41:39.543866 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.543880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.546251 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.547055 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.547066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.547853 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.548730 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.548760 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.549955 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.549959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.551811 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.551818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.552787 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.552790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.552997 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.554002 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.554020 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.556976 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.559569 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.559572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.559574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.560003 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.560330 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.560335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.560567 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.561074 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.561081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.561538 (+ 457us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.562374 (+ 836us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.562827 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":610,"spinlock_wait_cycles":50304,"thread_start_us":160,"threads_started":1}
I20260924 04:41:39.564061 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.542762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:50757)
0924 04:41:39.543383 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.543435 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.546238 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.547056 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.547067 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.547841 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.548740 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.548753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.549962 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.549968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.551814 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.551824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.552713 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.552719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.553007 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.553859 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.553897 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.556104 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.559352 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.559361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.559382 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.559971 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.560330 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.560333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.560335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.560567 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.561074 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.561081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.561538 (+ 457us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.562995 (+ 1457us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.563114 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":539,"spinlock_wait_cycles":35968,"thread_start_us":157,"threads_started":1}
I20260924 04:41:39.565433 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38757
I20260924 04:41:39.565450 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39651
I20260924 04:41:39.565433 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41423
I20260924 04:41:39.565711 7608 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.565711 7609 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.565711 7610 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.566311 7610 heartbeater.cc:511] Master 127.0.98.61:41423 requested a full tablet report, sending...
I20260924 04:41:39.566313 7608 heartbeater.cc:511] Master 127.0.98.60:39651 requested a full tablet report, sending...
I20260924 04:41:39.566311 7609 heartbeater.cc:511] Master 127.0.98.62:38757 requested a full tablet report, sending...
I20260924 04:41:39.567466 6507 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289)
I20260924 04:41:39.567734 7108 ts_manager.cc:194] Registered new tserver with Master: 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289)
I20260924 04:41:39.568300 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50757
I20260924 04:41:39.569068 7283 ts_manager.cc:194] Registered new tserver with Master: 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289)
W20260924 04:41:39.675721 7615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:39.675992 7615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:39.676036 7615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:39.680811 7615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:39.680884 7615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:39.680948 7615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:39.686617 7615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36965
--tserver_master_addrs=127.0.98.62:38757,127.0.98.61:41423,127.0.98.60:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:39.687763 7615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:39.688825 7615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:39.699666 7620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:39.699615 7621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.699736 7615 server_base.cc:1034] running on GCE node
W20260924 04:41:39.699607 7623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:39.700510 7615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:39.701254 7615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:39.702455 7615 hybrid_clock.cc:648] HybridClock initialized: now 1790224899702441 us; error 38 us; skew 500 ppm
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:39.705811 7615 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:41:39.706775 7615 webserver.cc:460] Webserver started at http://127.0.98.3:36965/ using document root <none> and password file <none>
I20260924 04:41:39.707448 7615 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:39.707564 7615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:39.711632 7615 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:41:39.713896 7630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.715312 7615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:39.715445 7615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "8fbb6f7977324456ab7632d01606d03e"
format_stamp: "Formatted at 2026-09-24 04:41:38 on dist-test-slave-npg2"
server_key: "53e875c26f587a1d8c9109519e43958f"
server_key_iv: "e47dfca11f12d8d89347c14daca7c17b"
server_key_version: "encryptionkey@0"
I20260924 04:41:39.716049 7615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:39.730638 7615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:39.733963 7615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:39.734189 7615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:39.734817 7615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:39.735806 7615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:39.735880 7615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.735955 7615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:39.735999 7615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:39.746013 7615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39157
I20260924 04:41:39.746034 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39157 every 8 connection(s)
I20260924 04:41:39.747157 7615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:39.750062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 7615
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:41:39 dist-test-slave-npg2 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:39.766494 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.749471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33779 (local address 127.0.98.60:39651)
0924 04:41:39.749632 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.749637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.750806 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 04:41:39.752071 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.752078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.752141 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.752199 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.755109 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.755943 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.757638 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.757934 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.758013 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.761458 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.761461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.761464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.761489 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.763785 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.764524 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.764528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.764533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.764617 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.765048 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.765054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.765058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.765300 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.765378 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.766150 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.766288 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:41:39.768301 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.751335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41993 (local address 127.0.98.62:38757)
0924 04:41:39.751476 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.751482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.754018 (+ 2536us) server_negotiation.cc:407] Connection header received
0924 04:41:39.754159 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.754161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.754233 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.754275 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.754884 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.755749 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.760002 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.760406 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.760502 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.764519 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.764525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.764527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.764553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.766457 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.766947 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.766953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.766957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.767042 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.767378 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.767381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.767383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.767595 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.767675 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.768005 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.768121 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:41:39.768390 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.751168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56863 (local address 127.0.98.61:41423)
0924 04:41:39.751297 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.751302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.751789 (+ 487us) server_negotiation.cc:407] Connection header received
0924 04:41:39.752061 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.752068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.752124 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.752188 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.755863 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.756610 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.757766 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.758107 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.758196 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.763595 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.763603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.763608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.763637 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.765777 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.766715 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.766717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.766719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.766802 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.767106 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.767109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.767111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.767351 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.767422 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.768085 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.768200 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:41:39.768533 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.749980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.98.3:33779)
0924 04:41:39.750646 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.750701 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.751814 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.752631 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.752640 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.753280 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.754524 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.754550 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.756122 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.756126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.757518 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.757528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.758238 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.758244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.758488 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.759220 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.759240 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.761273 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.763921 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.763929 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.763951 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.764399 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.764737 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.764741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.764743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.764938 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.765464 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.765473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.765839 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.766982 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.767377 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":3,"thread_start_us":132,"threads_started":1}
I20260924 04:41:39.768879 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.749980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:41993)
0924 04:41:39.753945 (+ 3965us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.753958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.754070 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.754395 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.754399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.754619 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.754747 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.754754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.758677 (+ 3923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.758683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.759651 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.759661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.760892 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.760897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.761016 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.761774 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.761793 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.764353 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.766566 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.766572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.766574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.766825 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.767182 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.767185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.767187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.767279 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.767743 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.767746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.767830 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.768531 (+ 701us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.768677 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3876,"mutex_wait_us":934,"thread_start_us":99,"threads_started":1}
I20260924 04:41:39.769459 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.750219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.98.3:56863)
0924 04:41:39.751680 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.751695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.751815 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.752641 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.752645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.754819 (+ 2174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:39.754956 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.754964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.756750 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.756753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.757632 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.757643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.758366 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.758369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.758492 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.760689 (+ 2197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:39.760705 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:39.763401 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:39.765895 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.765898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.765900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.766150 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.766918 (+ 768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:39.766923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:39.766925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:39.767011 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:39.767488 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:39.767490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:39.767569 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.768566 (+ 997us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.769292 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"mutex_wait_us":1755,"thread_start_us":133,"threads_started":1}
I20260924 04:41:39.769706 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.752909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.0.98.62:38757)
0924 04:41:39.753415 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.753419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.753720 (+ 301us) server_negotiation.cc:407] Connection header received
0924 04:41:39.753770 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.753774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.753828 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.753914 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.758679 (+ 4765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.759524 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.760875 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.761209 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.761309 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.762974 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.762981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.762984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.763013 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.766443 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.767295 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.767300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.767305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.767387 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.767674 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.767677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.767679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.767857 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.767981 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.769207 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.769343 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":67,"threads_started":1}
I20260924 04:41:39.769950 7744 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39651
I20260924 04:41:39.770200 7744 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.770604 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41423
I20260924 04:41:39.770680 7746 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.770812 7744 heartbeater.cc:511] Master 127.0.98.60:39651 requested a full tablet report, sending...
I20260924 04:41:39.770844 7746 heartbeater.cc:511] Master 127.0.98.61:41423 requested a full tablet report, sending...
I20260924 04:41:39.771845 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.752145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.0.98.61:41423)
0924 04:41:39.752742 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.752747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.753298 (+ 551us) server_negotiation.cc:407] Connection header received
0924 04:41:39.753338 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.753341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.753386 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.753462 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.754882 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.755749 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.756935 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.757286 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.757379 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.761075 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.761082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.761085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.761113 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.766563 (+ 5450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.770293 (+ 3730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.770297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.770298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.770380 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.770661 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.770664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.770666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.770859 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.770980 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.771569 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.771694 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":72,"threads_started":1}
I20260924 04:41:39.772037 7283 ts_manager.cc:194] Registered new tserver with Master: 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157)
I20260924 04:41:39.772045 7108 ts_manager.cc:194] Registered new tserver with Master: 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157)
I20260924 04:41:39.772219 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38757
I20260924 04:41:39.772320 7745 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:39.772485 7745 heartbeater.cc:511] Master 127.0.98.62:38757 requested a full tablet report, sending...
I20260924 04:41:39.773267 6507 ts_manager.cc:194] Re-registered known tserver with Master: 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157)
I20260924 04:41:39.773882 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.752175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.98.60:39651)
0924 04:41:39.753357 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.753362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.753375 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:41:39.753409 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.753413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.753458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.753545 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:39.755299 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.756013 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.757197 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.757524 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.757614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.762516 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:39.762523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:39.762532 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:39.762561 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:39.765012 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.768730 (+ 3718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.768733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.768735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.768820 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:39.769150 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:39.769155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:39.769157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:39.773104 (+ 3947us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:39.773224 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.773557 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.773688 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":101,"threads_started":1}
I20260924 04:41:39.774065 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41993
I20260924 04:41:39.773604 6506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:39.777455 6506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:39.780305 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: true } } }
I20260924 04:41:39.780459 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.784544 7763 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: attempting to promote NON_VOTER bf4120f6949c40b8a468142e68d644b4 to VOTER
I20260924 04:41:39.784947 7763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:39.786352 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 LEARNER]: Refusing update from remote peer 6bd3e0098d31455d999a95d5809cefa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:41:39.786474 7118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Refusing update from remote peer 6bd3e0098d31455d999a95d5809cefa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:41:39.786770 7764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:41:39.786906 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:41:39.788575 7762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bf4120f6949c40b8a468142e68d644b4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.789098 7118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bf4120f6949c40b8a468142e68d644b4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.790071 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.790194 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.790091 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.790355 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.790519 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.790601 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:39.790735 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.790872 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:39.790673 7295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bf4120f6949c40b8a468142e68d644b4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.793012 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bd3e0098d31455d999a95d5809cefa8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:39.793133 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:39.799547 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.793456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34639 (local address 127.0.0.1:53002)
0924 04:41:39.793596 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.793613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.793699 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.794793 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.794796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.794815 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:39.794932 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.794939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.796299 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.796305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.797517 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.797528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.799110 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.799114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.799231 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.799262 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.799316 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.799397 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:41:39.800289 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.793839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.0.98.1:34639)
0924 04:41:39.794210 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.794230 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.794255 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:41:39.794353 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.794359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.794596 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.794827 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:39.795081 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.796176 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.798037 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.798980 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.799818 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.799977 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.800030 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.800124 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":74,"threads_started":1}
I20260924 04:41:39.800613 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.793811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39157 (local address 127.0.0.1:43886)
0924 04:41:39.795068 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.795085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.795199 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.796397 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.796400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.796417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:39.796578 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.796584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.797900 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.797903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.798931 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.798940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.800130 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.800134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.800228 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.800309 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.800381 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.800462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"mutex_wait_us":30,"thread_start_us":111,"threads_started":1}
I20260924 04:41:39.801153 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.794213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.3:39157)
0924 04:41:39.795940 (+ 1727us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.795953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.795976 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:41:39.796053 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.796059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.796247 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.796414 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:39.796746 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.797777 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.799046 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.800009 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.800131 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.800276 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.800861 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.800984 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1608,"thread_start_us":92,"threads_started":1}
I20260924 04:41:39.801721 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.794069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35289 (local address 127.0.0.1:50314)
0924 04:41:39.796657 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:39.796673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:39.796761 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:39.797240 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:39.797243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:39.797261 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:39.797343 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.797349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.798825 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.798839 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.800093 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:39.800103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.801328 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.801331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:39.801424 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.801447 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:41:39.801528 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:41:39.801601 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2511,"thread_start_us":108,"threads_started":1}
I20260924 04:41:39.801815 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:39.794467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.0.98.2:35289)
0924 04:41:39.795473 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:39.795487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:39.796762 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 04:41:39.796875 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:39.796881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:39.797076 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:39.797233 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:39.797521 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.798679 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.800226 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:39.801007 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:39.801119 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:39.801257 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:39.801579 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 04:41:39.801677 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":94,"threads_started":1}
I20260924 04:41:39.803623 7402 tablet_service.cc:1467] Processing CreateTablet for tablet d24da7ba4ddf4b5489d818398df163f6 (DEFAULT_TABLE table=test-table [id=fb34fe76a69f47abb12b6315a405db18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:39.803957 7678 tablet_service.cc:1467] Processing CreateTablet for tablet d24da7ba4ddf4b5489d818398df163f6 (DEFAULT_TABLE table=test-table [id=fb34fe76a69f47abb12b6315a405db18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:39.804461 7542 tablet_service.cc:1467] Processing CreateTablet for tablet d24da7ba4ddf4b5489d818398df163f6 (DEFAULT_TABLE table=test-table [id=fb34fe76a69f47abb12b6315a405db18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:39.804915 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24da7ba4ddf4b5489d818398df163f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:39.804922 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24da7ba4ddf4b5489d818398df163f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:39.805421 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24da7ba4ddf4b5489d818398df163f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:39.812088 7778 tablet_bootstrap.cc:492] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Bootstrap starting.
I20260924 04:41:39.813614 7779 tablet_bootstrap.cc:492] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Bootstrap starting.
I20260924 04:41:39.815500 7780 tablet_bootstrap.cc:492] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Bootstrap starting.
I20260924 04:41:39.815812 7779 tablet_bootstrap.cc:654] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:39.815832 7778 tablet_bootstrap.cc:654] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:39.816540 7779 log.cc:824] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:39.816628 7778 log.cc:824] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:39.817600 7780 tablet_bootstrap.cc:654] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:39.818171 7780 log.cc:824] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:39.818652 7778 tablet_bootstrap.cc:492] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: No bootstrap required, opened a new log
I20260924 04:41:39.818652 7779 tablet_bootstrap.cc:492] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: No bootstrap required, opened a new log
I20260924 04:41:39.818848 7778 ts_tablet_manager.cc:1397] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:41:39.818848 7779 ts_tablet_manager.cc:1397] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:41:39.820142 7780 tablet_bootstrap.cc:492] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: No bootstrap required, opened a new log
I20260924 04:41:39.820312 7780 ts_tablet_manager.cc:1397] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:41:39.823659 7778 raft_consensus.cc:348] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:39.823659 7779 raft_consensus.cc:348] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:39.823932 7778 raft_consensus.cc:374] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:39.823932 7779 raft_consensus.cc:374] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:39.823984 7778 raft_consensus.cc:729] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbb6f7977324456ab7632d01606d03e, State: Initialized, Role: FOLLOWER
I20260924 04:41:39.823984 7779 raft_consensus.cc:729] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f8948fdd44b9590a13d124de15253, State: Initialized, Role: FOLLOWER
I20260924 04:41:39.824473 7778 consensus_queue.cc:260] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:39.824517 7779 consensus_queue.cc:260] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:39.824874 7780 raft_consensus.cc:348] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:39.825115 7780 raft_consensus.cc:374] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:39.825157 7780 raft_consensus.cc:729] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faf10a6f2b641e886914ca2c67a591c, State: Initialized, Role: FOLLOWER
I20260924 04:41:39.825495 7778 ts_tablet_manager.cc:1428] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:41:39.825516 7779 ts_tablet_manager.cc:1428] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:41:39.825536 7780 consensus_queue.cc:260] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:39.825876 7745 heartbeater.cc:503] Master 127.0.98.62:38757 was elected leader, sending a full tablet report...
I20260924 04:41:39.826418 7780 ts_tablet_manager.cc:1428] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:41:39.826741 7472 heartbeater.cc:503] Master 127.0.98.62:38757 was elected leader, sending a full tablet report...
I20260924 04:41:39.831377 7609 heartbeater.cc:503] Master 127.0.98.62:38757 was elected leader, sending a full tablet report...
W20260924 04:41:39.998991 7747 tablet.cc:2369] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:40.020454 7786 raft_consensus.cc:482] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:40.020735 7786 raft_consensus.cc:504] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:40.021973 7786 leader_election.cc:288] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157), 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289)
I20260924 04:41:40.027225 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.022373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32899 (local address 127.0.98.2:35289)
0924 04:41:40.022563 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.022568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.022645 (+ 77us) server_negotiation.cc:407] Connection header received
0924 04:41:40.022802 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.022805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.022857 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.022943 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:40.023440 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.024339 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.026186 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.026660 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.026751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.026852 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.027003 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.027079 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 04:41:40.027222 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.022240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35289 (local address 127.0.98.1:32899)
0924 04:41:40.022524 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:40.022540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:40.022674 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:40.022971 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:40.022974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:40.023001 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:40.023250 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:40.023256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.024459 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.024491 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:40.026021 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:40.026033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.026785 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.026789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:40.026866 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.026888 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:41:40.026956 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:41:40.027017 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":117,"spinlock_wait_cycles":17536,"thread_start_us":119,"threads_started":1}
I20260924 04:41:40.028023 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.022245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39157 (local address 127.0.98.1:49031)
0924 04:41:40.022555 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:40.022570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:40.022674 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:40.022971 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:40.022974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:40.023001 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:40.023249 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:40.023256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.024551 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.024554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:40.026021 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:40.026033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.027608 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.027611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:40.027710 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.027733 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:41:40.027838 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:41:40.027909 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":252,"spinlock_wait_cycles":7296,"thread_start_us":78,"threads_started":1}
I20260924 04:41:40.028055 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.022505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49031 (local address 127.0.98.3:39157)
0924 04:41:40.022628 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.022634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.022649 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:41:40.022802 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.022805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.022856 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.022959 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:40.023425 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.024439 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.026894 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.027495 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.027580 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.027624 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.027863 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.027942 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:41:40.028023 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24da7ba4ddf4b5489d818398df163f6" candidate_uuid: "5faf10a6f2b641e886914ca2c67a591c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f8948fdd44b9590a13d124de15253" is_pre_election: true
I20260924 04:41:40.028359 7562 raft_consensus.cc:2454] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5faf10a6f2b641e886914ca2c67a591c in term 0.
I20260924 04:41:40.028877 7361 leader_election.cc:302] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5faf10a6f2b641e886914ca2c67a591c, 8d2f8948fdd44b9590a13d124de15253; no voters:
I20260924 04:41:40.029163 7786 raft_consensus.cc:2781] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:40.029251 7786 raft_consensus.cc:482] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:40.029183 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24da7ba4ddf4b5489d818398df163f6" candidate_uuid: "5faf10a6f2b641e886914ca2c67a591c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbb6f7977324456ab7632d01606d03e" is_pre_election: true
I20260924 04:41:40.029311 7786 raft_consensus.cc:3037] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:40.029470 7698 raft_consensus.cc:2454] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5faf10a6f2b641e886914ca2c67a591c in term 0.
I20260924 04:41:40.030925 7786 raft_consensus.cc:504] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:40.031358 7786 leader_election.cc:288] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [CANDIDATE]: Term 1 election: Requested vote from peers 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157), 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289)
I20260924 04:41:40.031700 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24da7ba4ddf4b5489d818398df163f6" candidate_uuid: "5faf10a6f2b641e886914ca2c67a591c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbb6f7977324456ab7632d01606d03e"
I20260924 04:41:40.031739 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24da7ba4ddf4b5489d818398df163f6" candidate_uuid: "5faf10a6f2b641e886914ca2c67a591c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f8948fdd44b9590a13d124de15253"
I20260924 04:41:40.031844 7698 raft_consensus.cc:3037] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:40.031867 7562 raft_consensus.cc:3037] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:40.032881 7562 raft_consensus.cc:2454] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5faf10a6f2b641e886914ca2c67a591c in term 1.
I20260924 04:41:40.032912 7698 raft_consensus.cc:2454] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5faf10a6f2b641e886914ca2c67a591c in term 1.
I20260924 04:41:40.033156 7361 leader_election.cc:302] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5faf10a6f2b641e886914ca2c67a591c, 8d2f8948fdd44b9590a13d124de15253; no voters:
I20260924 04:41:40.033357 7786 raft_consensus.cc:2781] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:40.033531 7786 raft_consensus.cc:686] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 1 LEADER]: Becoming Leader. State: Replica: 5faf10a6f2b641e886914ca2c67a591c, State: Running, Role: LEADER
I20260924 04:41:40.033928 7786 consensus_queue.cc:237] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } }
I20260924 04:41:40.037528 6506 catalog_manager.cc:5495] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5faf10a6f2b641e886914ca2c67a591c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } health_report { overall_health: UNKNOWN } } }
W20260924 04:41:40.042219 7611 tablet.cc:2369] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:40.067533 7474 tablet.cc:2369] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:40.092864 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.089033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.0.98.1:34639)
0924 04:41:40.089175 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.089182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.089201 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:41:40.089294 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.089296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.089353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.089474 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:40.089766 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.090325 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.091272 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.091500 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.091620 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.091980 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:40.092433 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:40.092597 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.092686 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.092747 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:41:40.099457 7698 raft_consensus.cc:1261] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Refusing update from remote peer 5faf10a6f2b641e886914ca2c67a591c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:40.099457 7562 raft_consensus.cc:1261] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Refusing update from remote peer 5faf10a6f2b641e886914ca2c67a591c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:40.100248 7789 consensus_queue.cc:1035] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:40.100477 7786 consensus_queue.cc:1035] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:40.111184 7793 mvcc.cc:205] Tried to move back new op lower bound from 7332761190799327232 to 7332761190541611008. Current Snapshot: MvccSnapshot[applied={T|T < 7332761190799327232}]
I20260924 04:41:40.118346 7794 mvcc.cc:205] Tried to move back new op lower bound from 7332761190799327232 to 7332761190541611008. Current Snapshot: MvccSnapshot[applied={T|T < 7332761190799327232 or (T in {7332761190799327232})}]
I20260924 04:41:40.119334 6507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:40.119480 6507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:40.125342 6507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bd3e0098d31455d999a95d5809cefa8: Sending DeleteTablet for 3 replicas of tablet d24da7ba4ddf4b5489d818398df163f6
I20260924 04:41:40.126189 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet d24da7ba4ddf4b5489d818398df163f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:40 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50314
I20260924 04:41:40.126189 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet d24da7ba4ddf4b5489d818398df163f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:40 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43886
I20260924 04:41:40.126230 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet d24da7ba4ddf4b5489d818398df163f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:40 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53002
I20260924 04:41:40.126717 7802 tablet_replica.cc:330] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: stopping tablet replica
I20260924 04:41:40.127032 7803 tablet_replica.cc:330] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: stopping tablet replica
I20260924 04:41:40.127085 7802 raft_consensus.cc:2229] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:40.127471 7802 raft_consensus.cc:2258] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:40.127947 7803 raft_consensus.cc:2229] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:40.128293 7803 raft_consensus.cc:2258] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:40.129314 7801 tablet_replica.cc:330] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: stopping tablet replica
I20260924 04:41:40.129781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6475
I20260924 04:41:40.130632 7803 ts_tablet_manager.cc:1905] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:40.132156 7802 ts_tablet_manager.cc:1905] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:40.133647 7803 ts_tablet_manager.cc:1918] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:40.133766 7803 log.cc:1196] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/d24da7ba4ddf4b5489d818398df163f6
I20260924 04:41:40.134192 7803 ts_tablet_manager.cc:1939] T d24da7ba4ddf4b5489d818398df163f6 P 5faf10a6f2b641e886914ca2c67a591c: Deleting consensus metadata
I20260924 04:41:40.135300 7802 ts_tablet_manager.cc:1918] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:40.135473 7802 log.cc:1196] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/d24da7ba4ddf4b5489d818398df163f6
I20260924 04:41:40.135857 7802 ts_tablet_manager.cc:1939] T d24da7ba4ddf4b5489d818398df163f6 P 8fbb6f7977324456ab7632d01606d03e: Deleting consensus metadata
W20260924 04:41:40.138093 7632 connection.cc:535] server connection from 127.0.0.1:43886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 04:41:40.138105 7360 connection.cc:535] server connection from 127.0.0.1:53002 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:41:40.141525 7801 raft_consensus.cc:2229] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:40.141898 7801 raft_consensus.cc:2258] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:40.143994 7801 ts_tablet_manager.cc:1905] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:40.146737 7801 ts_tablet_manager.cc:1918] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:40.146844 7801 log.cc:1196] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/d24da7ba4ddf4b5489d818398df163f6
I20260924 04:41:40.147183 7801 ts_tablet_manager.cc:1939] T d24da7ba4ddf4b5489d818398df163f6 P 8d2f8948fdd44b9590a13d124de15253: Deleting consensus metadata
W20260924 04:41:40.148373 7497 connection.cc:441] server connection from 127.0.0.1:50314 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50314 (request call id 1) could send its response
I20260924 04:41:40.152275 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.128248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.0.98.61:41423)
0924 04:41:40.128402 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.128407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.139444 (+ 11037us) server_negotiation.cc:407] Connection header received
0924 04:41:40.139648 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.139652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.139714 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.139812 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:40.141127 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.142273 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.143995 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.144479 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.144590 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.146618 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:40.146623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:40.146626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:40.146654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:40.148972 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.149581 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.149586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.149589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.149683 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.150014 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.150021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.150026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.150307 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:40.150453 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.151923 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.152061 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:41:40.155043 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.142377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.0.98.61:41423)
0924 04:41:40.142520 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.142525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.143589 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 04:41:40.143760 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.143777 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.143827 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.143888 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:40.145010 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.145892 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.147432 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.147875 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.147973 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.149792 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:40.149796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:40.149799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:40.149825 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:40.151916 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.152856 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.152863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.152866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.153274 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.154041 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.154047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.154050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.154284 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:40.154391 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.154750 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.154882 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:41:40.158171 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.142673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.98.60:39651)
0924 04:41:40.143249 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.143254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.148785 (+ 5531us) server_negotiation.cc:407] Connection header received
0924 04:41:40.148960 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.148963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.149014 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.149067 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:40.149991 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.150803 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.152063 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.152398 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.152471 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.154296 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:40.154300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:40.154302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:40.154330 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:40.156188 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.156791 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.156794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.156796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.156883 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.157196 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.157199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.157201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.157401 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:40.157501 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.157885 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.158016 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
I20260924 04:41:40.161887 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:40.128537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.98.60:39651)
0924 04:41:40.128754 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:40.128759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:40.151536 (+ 22777us) server_negotiation.cc:407] Connection header received
0924 04:41:40.151585 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:40.151590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:40.151653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:40.151739 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:40.154197 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.155037 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.156206 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:40.156578 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:40.156714 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:40.158128 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:40.158134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:40.158142 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:40.158173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:40.159998 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.160495 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.160501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.160503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.160591 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:40.160930 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:40.160933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:40.160935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:40.161117 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:40.161259 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:40.161526 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 04:41:40.161658 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260924 04:41:41.113140 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.112614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:53309)
0924 04:41:41.112947 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.113039 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":91,"threads_started":1}
W20260924 04:41:41.113541 7609 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.117893 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.117441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:54685)
0924 04:41:41.117679 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.117807 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
W20260924 04:41:41.118345 7745 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.120057 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.119574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:55985)
0924 04:41:41.119895 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.119973 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":70,"threads_started":1}
W20260924 04:41:41.120536 7472 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.624605 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bd3e0098d31455d999a95d5809cefa8)
I20260924 04:41:41.624864 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:41.624931 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bd3e0098d31455d999a95d5809cefa8)
I20260924 04:41:41.625115 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:41.626047 7815 leader_election.cc:288] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757), 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61:41423)
I20260924 04:41:41.626381 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757), bf4120f6949c40b8a468142e68d644b4 (127.0.98.60:39651)
I20260924 04:41:41.626806 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.626183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42508)
0924 04:41:41.626590 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.626679 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":78,"threads_started":1}
I20260924 04:41:41.626971 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.626192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42506)
0924 04:41:41.626810 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.626877 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":77,"threads_started":1}
W20260924 04:41:41.627100 7092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
W20260924 04:41:41.627099 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.628044 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.627741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42522)
0924 04:41:41.627935 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.627981 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 04:41:41.628054 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.627808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42530)
0924 04:41:41.627935 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.627984 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 04:41:41.628254 7092 leader_election.cc:334] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
W20260924 04:41:41.628291 7276 leader_election.cc:334] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.631582 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.626183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39651 (local address 127.0.0.1:33752)
0924 04:41:41.626505 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.626520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:41.626626 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:41.627649 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:41.627652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:41.627672 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:41.627823 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:41.627831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.628895 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.628899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:41.630187 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:41.630197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.631069 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.631073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:41.631256 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:41.631277 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:41:41.631348 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:41:41.631447 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":21,"spinlock_wait_cycles":3968,"thread_start_us":66,"threads_started":1}
I20260924 04:41:41.631605 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.626522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:44558)
0924 04:41:41.626951 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.626963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:41.627529 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:41.627950 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:41.627953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:41.627971 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:41.628171 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:41.628181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.629073 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.629075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:41.630243 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:41.630255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.631069 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.631073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:41.631236 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:41.631260 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:41:41.631347 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:41:41.631435 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":68,"spinlock_wait_cycles":976128,"thread_start_us":61,"threads_started":1}
I20260924 04:41:41.632138 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.626691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.0.98.60:39651)
0924 04:41:41.626882 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:41.626886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:41.626900 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:41.626949 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:41.626952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:41.627508 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:41.627927 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:41.628070 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.628795 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.630341 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.630945 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.631055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:41.631171 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:41.631373 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:41:41.631467 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":130,"thread_start_us":52,"threads_started":1}
I20260924 04:41:41.632256 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.626606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.98.61:41423)
0924 04:41:41.626836 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:41.626842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:41.627023 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:41:41.627682 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:41.627687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:41.627756 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:41.627814 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:41.628301 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.628906 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.630366 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:41.630953 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:41.631055 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:41.631208 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:41.631373 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:41:41.631562 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":164,"spinlock_wait_cycles":1536,"thread_start_us":50,"threads_started":1}
I20260924 04:41:41.632488 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6107f15bf1c14dffb36a132e8e566eca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bf4120f6949c40b8a468142e68d644b4" is_pre_election: true
I20260924 04:41:41.632681 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6107f15bf1c14dffb36a132e8e566eca in term 1.
I20260924 04:41:41.632707 7118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4120f6949c40b8a468142e68d644b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6107f15bf1c14dffb36a132e8e566eca" is_pre_election: true
I20260924 04:41:41.632860 7118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4120f6949c40b8a468142e68d644b4 in term 1.
I20260924 04:41:41.633075 7093 leader_election.cc:302] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6107f15bf1c14dffb36a132e8e566eca, bf4120f6949c40b8a468142e68d644b4; no voters: 6bd3e0098d31455d999a95d5809cefa8
I20260924 04:41:41.633262 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:41:41.633349 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bd3e0098d31455d999a95d5809cefa8)
I20260924 04:41:41.633406 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:41:41.633512 7278 leader_election.cc:302] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6107f15bf1c14dffb36a132e8e566eca, bf4120f6949c40b8a468142e68d644b4; no voters: 6bd3e0098d31455d999a95d5809cefa8
I20260924 04:41:41.633810 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:41:41.633877 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bd3e0098d31455d999a95d5809cefa8)
I20260924 04:41:41.633929 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:41:41.634538 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:41.634931 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [CANDIDATE]: Term 2 election: Requested vote from peers 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757), bf4120f6949c40b8a468142e68d644b4 (127.0.98.60:39651)
I20260924 04:41:41.634891 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:41.635236 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.635012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42536)
0924 04:41:41.635098 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.635174 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 04:41:41.635273 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6107f15bf1c14dffb36a132e8e566eca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bf4120f6949c40b8a468142e68d644b4"
I20260924 04:41:41.635418 7295 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6107f15bf1c14dffb36a132e8e566eca in current term 2: Already voted for candidate bf4120f6949c40b8a468142e68d644b4 in this term.
I20260924 04:41:41.635465 7815 leader_election.cc:288] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 2 election: Requested vote from peers 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757), 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61:41423)
I20260924 04:41:41.635620 7118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4120f6949c40b8a468142e68d644b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6107f15bf1c14dffb36a132e8e566eca"
I20260924 04:41:41.635769 7118 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bf4120f6949c40b8a468142e68d644b4 in current term 2: Already voted for candidate 6107f15bf1c14dffb36a132e8e566eca in this term.
I20260924 04:41:41.635870 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.635507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42538)
0924 04:41:41.635754 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.635816 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260924 04:41:41.636440 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.636216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42540)
0924 04:41:41.636324 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.636379 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 04:41:41.636459 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:41.636267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42556)
0924 04:41:41.636375 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:41.636415 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:41:41.636615 7092 leader_election.cc:334] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.636699 7092 leader_election.cc:302] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6107f15bf1c14dffb36a132e8e566eca; no voters: 6bd3e0098d31455d999a95d5809cefa8, bf4120f6949c40b8a468142e68d644b4
I20260924 04:41:41.636828 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 04:41:41.636938 7276 leader_election.cc:334] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:41.637005 7276 leader_election.cc:302] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf4120f6949c40b8a468142e68d644b4; no voters: 6107f15bf1c14dffb36a132e8e566eca, 6bd3e0098d31455d999a95d5809cefa8
I20260924 04:41:41.637108 7815 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:41:42.114800 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:42.114330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:47253)
0924 04:41:42.114620 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:42.114690 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260924 04:41:42.119429 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:42.118945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:45957)
0924 04:41:42.119235 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:42.119324 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260924 04:41:42.121429 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:42.121072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:51875)
0924 04:41:42.121290 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:42.121353 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260924 04:41:43.116179 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.115726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:40209)
0924 04:41:43.116013 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.116082 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260924 04:41:43.120434 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.120004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:51993)
0924 04:41:43.120248 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.120329 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260924 04:41:43.122434 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.122096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:43741)
0924 04:41:43.122305 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.122360 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":54,"threads_started":1}
I20260924 04:41:43.239100 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:43.239512 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:43.240025 7831 leader_election.cc:288] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757), 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61:41423)
I20260924 04:41:43.240587 7118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4120f6949c40b8a468142e68d644b4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6107f15bf1c14dffb36a132e8e566eca" is_pre_election: true
I20260924 04:41:43.240798 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.240329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42764)
0924 04:41:43.240606 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.240665 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
I20260924 04:41:43.240794 7118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4120f6949c40b8a468142e68d644b4 in term 2.
I20260924 04:41:43.241169 7278 leader_election.cc:302] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6107f15bf1c14dffb36a132e8e566eca, bf4120f6949c40b8a468142e68d644b4; no voters:
I20260924 04:41:43.241366 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 04:41:43.241428 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:43.241483 7831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:41:43.241773 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.241588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42766)
0924 04:41:43.241672 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.241717 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 04:41:43.241909 7276 leader_election.cc:334] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:43.242658 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:43.242988 7831 leader_election.cc:288] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 3 election: Requested vote from peers 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757), 6107f15bf1c14dffb36a132e8e566eca (127.0.98.61:41423)
I20260924 04:41:43.243301 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.243056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42782)
0924 04:41:43.243165 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.243218 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 04:41:43.243386 7118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf4120f6949c40b8a468142e68d644b4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6107f15bf1c14dffb36a132e8e566eca"
I20260924 04:41:43.243494 7118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:41:43.243997 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.243794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42794)
0924 04:41:43.243899 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.243945 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:41:43.244107 7276 leader_election.cc:334] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
I20260924 04:41:43.244656 7118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4120f6949c40b8a468142e68d644b4 in term 3.
I20260924 04:41:43.244946 7278 leader_election.cc:302] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6107f15bf1c14dffb36a132e8e566eca, bf4120f6949c40b8a468142e68d644b4; no voters: 6bd3e0098d31455d999a95d5809cefa8
I20260924 04:41:43.245096 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:41:43.245287 7831 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [term 3 LEADER]: Becoming Leader. State: Replica: bf4120f6949c40b8a468142e68d644b4, State: Running, Role: LEADER
I20260924 04:41:43.245548 7831 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } }
I20260924 04:41:43.246869 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4120f6949c40b8a468142e68d644b4. Latest consensus state: current_term: 3 leader_uuid: "bf4120f6949c40b8a468142e68d644b4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:43.247020 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:43.247334 7836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:43.249543 7836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:43.249953 7836 catalog_manager.cc:1259] Loaded cluster ID: d287a7371e6b4378aa998f2f500332a8
I20260924 04:41:43.249996 7836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:43.250725 7836 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:43.251209 7836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Loaded TSK: 0
I20260924 04:41:43.252019 7836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:41:43.313122 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:43.316578 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:43.319748 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.319448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42808)
0924 04:41:43.319603 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.319655 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:41:43.319842 7118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [term 3 FOLLOWER]: Refusing update from remote peer bf4120f6949c40b8a468142e68d644b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 04:41:43.320322 7831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 04:41:43.320436 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.320224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:42818)
0924 04:41:43.320330 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.320375 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:41:43.320586 7276 consensus_peers.cc:487] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 -> Peer 6bd3e0098d31455d999a95d5809cefa8 (127.0.98.62:38757): Couldn't send request to peer 6bd3e0098d31455d999a95d5809cefa8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:41:43.321792 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4120f6949c40b8a468142e68d644b4. Latest consensus state: current_term: 3 leader_uuid: "bf4120f6949c40b8a468142e68d644b4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:43.321985 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:43.322985 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bf4120f6949c40b8a468142e68d644b4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:43.323150 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:43.323455 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bf4120f6949c40b8a468142e68d644b4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6bd3e0098d31455d999a95d5809cefa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } peers { permanent_uuid: "6107f15bf1c14dffb36a132e8e566eca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attrs { promote: false } } peers { permanent_uuid: "bf4120f6949c40b8a468142e68d644b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39651 } attrs { promote: false } } }
I20260924 04:41:43.323598 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6107f15bf1c14dffb36a132e8e566eca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:41:43.323724 7839 mvcc.cc:205] Tried to move back new op lower bound from 7332761203992514560 to 7332761203695898624. Current Snapshot: MvccSnapshot[applied={T|T < 7332761203992514560}]
I20260924 04:41:43.334429 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.330050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35289 (local address 127.0.0.1:54294)
0924 04:41:43.330222 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.330240 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:43.330336 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:43.331169 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:43.331172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:43.331189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:43.331288 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.331296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.332311 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.332315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.333217 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.333226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.334079 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.334082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.334188 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.334214 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:41:43.334268 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:41:43.334325 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 04:41:43.334906 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.330473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39157 (local address 127.0.0.1:37510)
0924 04:41:43.330712 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.330726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:43.330826 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:43.331511 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:43.331514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:43.331529 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:43.331658 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.331663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.332425 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.332428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.333618 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.333626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.334537 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.334543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.334625 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.334645 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:41:43.334696 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:41:43.334758 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":23,"thread_start_us":110,"threads_started":1}
I20260924 04:41:43.335047 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.330166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.0.98.2:35289)
0924 04:41:43.330773 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:43.330779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:43.330795 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:41:43.330845 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:43.330848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:43.330897 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:43.331022 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:43.331400 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.332196 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.333348 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.333951 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.334717 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.334775 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:43.334813 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 04:41:43.334910 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":86,"threads_started":1}
I20260924 04:41:43.335197 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.330847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.0.98.1:34639)
0924 04:41:43.331186 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:43.331190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:43.331204 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:43.331310 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:43.331313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:43.331359 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:43.331433 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:43.331827 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.332551 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.333881 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.334490 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.334581 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.334643 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:43.334905 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 04:41:43.335004 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":72,"threads_started":1}
I20260924 04:41:43.335074 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.330506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34639 (local address 127.0.0.1:48952)
0924 04:41:43.331090 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.331103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:43.331227 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:43.331537 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:43.331539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:43.331556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:43.331696 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.331702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.332677 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.332680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.333764 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.333771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.334708 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.334712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.334784 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.334803 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:41:43.334873 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:41:43.334939 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":22,"spinlock_wait_cycles":9472,"thread_start_us":97,"threads_started":1}
I20260924 04:41:43.335695 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 (DEFAULT_TABLE table=test-table [id=7d4e3fc8f7174ed4885e0b8c666f8ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:43.335863 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.330559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.0.98.3:39157)
0924 04:41:43.331296 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:43.331301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:43.331316 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:41:43.331368 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:43.331373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:43.331420 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:43.331495 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:43.331775 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.332320 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.333756 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.334381 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.335547 (+ 1166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.335610 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:43.335647 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 04:41:43.335727 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":65,"threads_started":1}
I20260924 04:41:43.336016 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:43.336177 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 (DEFAULT_TABLE table=test-table [id=7d4e3fc8f7174ed4885e0b8c666f8ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:43.336367 7678 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 (DEFAULT_TABLE table=test-table [id=7d4e3fc8f7174ed4885e0b8c666f8ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:43.336434 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:43.336588 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:43.338143 7849 tablet_bootstrap.cc:492] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Bootstrap starting.
I20260924 04:41:43.339100 7851 tablet_bootstrap.cc:492] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Bootstrap starting.
I20260924 04:41:43.339237 7849 tablet_bootstrap.cc:654] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:43.339547 7850 tablet_bootstrap.cc:492] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c: Bootstrap starting.
I20260924 04:41:43.340616 7849 tablet_bootstrap.cc:492] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: No bootstrap required, opened a new log
I20260924 04:41:43.340644 7851 tablet_bootstrap.cc:654] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:43.340693 7849 ts_tablet_manager.cc:1397] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 04:41:43.341255 7849 raft_consensus.cc:348] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.341389 7849 raft_consensus.cc:374] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:43.341428 7849 raft_consensus.cc:729] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f8948fdd44b9590a13d124de15253, State: Initialized, Role: FOLLOWER
I20260924 04:41:43.341571 7850 tablet_bootstrap.cc:654] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:43.341565 7849 consensus_queue.cc:260] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.341962 7849 ts_tablet_manager.cc:1428] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 04:41:43.342463 7851 tablet_bootstrap.cc:492] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: No bootstrap required, opened a new log
I20260924 04:41:43.342710 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.342408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:59349)
0924 04:41:43.342549 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.342619 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:41:43.342965 7850 tablet_bootstrap.cc:492] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c: No bootstrap required, opened a new log
I20260924 04:41:43.343086 7850 ts_tablet_manager.cc:1397] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:41:43.343521 7850 raft_consensus.cc:348] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.343742 7851 ts_tablet_manager.cc:1397] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 04:41:43.344192 7851 raft_consensus.cc:348] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.344292 7608 heartbeater.cc:503] Master 127.0.98.60:39651 was elected leader, sending a full tablet report...
I20260924 04:41:43.344355 7851 raft_consensus.cc:374] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:43.344393 7851 raft_consensus.cc:729] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbb6f7977324456ab7632d01606d03e, State: Initialized, Role: FOLLOWER
I20260924 04:41:43.344646 7850 raft_consensus.cc:374] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:43.344738 7850 raft_consensus.cc:729] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faf10a6f2b641e886914ca2c67a591c, State: Initialized, Role: FOLLOWER
I20260924 04:41:43.344528 7851 consensus_queue.cc:260] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.345068 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.344785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:37615)
0924 04:41:43.344937 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.344989 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 04:41:43.345265 7851 ts_tablet_manager.cc:1428] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:41:43.346551 7850 consensus_queue.cc:260] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.346959 7850 ts_tablet_manager.cc:1428] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260924 04:41:43.345734 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.345517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:56609)
0924 04:41:43.345609 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.345664 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:41:43.347349 7744 heartbeater.cc:503] Master 127.0.98.60:39651 was elected leader, sending a full tablet report...
I20260924 04:41:43.347755 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.347530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:38421)
0924 04:41:43.347632 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.347686 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:41:43.347908 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.347492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:54891)
0924 04:41:43.347748 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.347813 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260924 04:41:43.348965 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.348706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:60831)
0924 04:41:43.348849 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.348895 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 04:41:43.349390 7471 heartbeater.cc:503] Master 127.0.98.60:39651 was elected leader, sending a full tablet report...
I20260924 04:41:43.366816 7854 raft_consensus.cc:482] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:43.366958 7854 raft_consensus.cc:504] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.368170 7854 leader_election.cc:288] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157), 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639)
I20260924 04:41:43.372578 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.368606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59075 (local address 127.0.98.1:34639)
0924 04:41:43.368764 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:43.368769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:43.368905 (+ 136us) server_negotiation.cc:407] Connection header received
0924 04:41:43.369040 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:43.369042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:43.369081 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:43.369138 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:43.369414 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.370135 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.371531 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.371953 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.372066 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.372133 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:43.372366 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:41:43.372443 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:41:43.372596 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.368454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46293 (local address 127.0.98.3:39157)
0924 04:41:43.368596 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:43.368602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:43.368622 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:41:43.368756 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:43.368764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:43.368815 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:43.368886 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:43.369268 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.370067 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.371568 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.371963 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.372098 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.372179 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:43.372399 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:41:43.372477 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:41:43.372614 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.368451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34639 (local address 127.0.98.2:59075)
0924 04:41:43.368800 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.368820 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:43.368946 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:43.369181 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:43.369185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:43.369208 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:43.369302 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.369306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.370253 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.370256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.371396 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.371407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.372102 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.372106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.372238 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.372257 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:41:43.372318 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:41:43.372429 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":70,"thread_start_us":132,"threads_started":1}
I20260924 04:41:43.372627 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.368346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39157 (local address 127.0.98.2:46293)
0924 04:41:43.368472 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.368489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:43.368593 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:43.368958 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:43.368962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:43.368980 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:43.369111 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.369138 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.370203 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.370208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.371407 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:43.371417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.372110 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.372113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:43.372251 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.372269 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:41:43.372364 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:41:43.372443 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":100}
I20260924 04:41:43.373220 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7a1f3a94c142aeafbd1d8abdad57b9" candidate_uuid: "8d2f8948fdd44b9590a13d124de15253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbb6f7977324456ab7632d01606d03e" is_pre_election: true
I20260924 04:41:43.373296 7425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7a1f3a94c142aeafbd1d8abdad57b9" candidate_uuid: "8d2f8948fdd44b9590a13d124de15253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5faf10a6f2b641e886914ca2c67a591c" is_pre_election: true
I20260924 04:41:43.373383 7698 raft_consensus.cc:2454] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f8948fdd44b9590a13d124de15253 in term 0.
I20260924 04:41:43.373576 7425 raft_consensus.cc:2454] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f8948fdd44b9590a13d124de15253 in term 0.
I20260924 04:41:43.373873 7496 leader_election.cc:302] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2f8948fdd44b9590a13d124de15253, 8fbb6f7977324456ab7632d01606d03e; no voters:
I20260924 04:41:43.374109 7854 raft_consensus.cc:2781] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:43.374166 7854 raft_consensus.cc:482] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:43.374202 7854 raft_consensus.cc:3037] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:43.375236 7854 raft_consensus.cc:504] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.375588 7854 leader_election.cc:288] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [CANDIDATE]: Term 1 election: Requested vote from peers 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157), 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639)
I20260924 04:41:43.375885 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7a1f3a94c142aeafbd1d8abdad57b9" candidate_uuid: "8d2f8948fdd44b9590a13d124de15253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbb6f7977324456ab7632d01606d03e"
I20260924 04:41:43.375986 7698 raft_consensus.cc:3037] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:43.375976 7425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7a1f3a94c142aeafbd1d8abdad57b9" candidate_uuid: "8d2f8948fdd44b9590a13d124de15253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5faf10a6f2b641e886914ca2c67a591c"
I20260924 04:41:43.376112 7425 raft_consensus.cc:3037] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:43.377008 7698 raft_consensus.cc:2454] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f8948fdd44b9590a13d124de15253 in term 1.
I20260924 04:41:43.377054 7425 raft_consensus.cc:2454] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f8948fdd44b9590a13d124de15253 in term 1.
I20260924 04:41:43.377278 7496 leader_election.cc:302] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2f8948fdd44b9590a13d124de15253, 8fbb6f7977324456ab7632d01606d03e; no voters:
I20260924 04:41:43.377442 7854 raft_consensus.cc:2781] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:43.377681 7854 raft_consensus.cc:686] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 1 LEADER]: Becoming Leader. State: Replica: 8d2f8948fdd44b9590a13d124de15253, State: Running, Role: LEADER
I20260924 04:41:43.378150 7854 consensus_queue.cc:237] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } }
I20260924 04:41:43.378409 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.378151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:53729)
0924 04:41:43.378270 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.378328 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:41:43.380614 7283 catalog_manager.cc:5495] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8d2f8948fdd44b9590a13d124de15253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f8948fdd44b9590a13d124de15253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:43.403239 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.399507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.0.98.2:35289)
0924 04:41:43.399627 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:43.399632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:43.399646 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:43.399912 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:43.399923 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:43.400004 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:43.400130 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:43.400387 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.400908 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.401769 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:43.402000 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:43.402114 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:43.402387 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:43.402837 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:43.402968 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:43.403016 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:41:43.403070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:41:43.406139 7698 raft_consensus.cc:1261] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Refusing update from remote peer 8d2f8948fdd44b9590a13d124de15253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:43.406430 7425 raft_consensus.cc:1261] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Refusing update from remote peer 8d2f8948fdd44b9590a13d124de15253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:43.406733 7854 consensus_queue.cc:1035] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fbb6f7977324456ab7632d01606d03e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:43.407060 7859 consensus_queue.cc:1035] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5faf10a6f2b641e886914ca2c67a591c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:43.408367 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.408067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:59803)
0924 04:41:43.408219 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.408289 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 04:41:43.410140 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.409845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:44181)
0924 04:41:43.409999 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.410057 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 04:41:43.412436 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.412156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.2:46695)
0924 04:41:43.412306 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.412361 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 04:41:43.412588 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.412135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.3:46263)
0924 04:41:43.412466 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.412519 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
I20260924 04:41:43.414738 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.414481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.98.1:46735)
0924 04:41:43.414623 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.414671 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 04:41:43.415625 7865 mvcc.cc:205] Tried to move back new op lower bound from 7332761204346413056 to 7332761204239503360. Current Snapshot: MvccSnapshot[applied={T|T < 7332761204346413056}]
I20260924 04:41:43.419796 7283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:43.419961 7283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:43.424077 7283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf4120f6949c40b8a468142e68d644b4: Sending DeleteTablet for 3 replicas of tablet 9e7a1f3a94c142aeafbd1d8abdad57b9
I20260924 04:41:43.424708 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54294
I20260924 04:41:43.424937 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37510
I20260924 04:41:43.425202 7872 tablet_replica.cc:330] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: stopping tablet replica
I20260924 04:41:43.425221 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48952
I20260924 04:41:43.425446 7872 raft_consensus.cc:2229] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:43.425653 7874 tablet_replica.cc:330] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c: stopping tablet replica
I20260924 04:41:43.425813 7874 raft_consensus.cc:2229] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:43.425868 7873 tablet_replica.cc:330] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: stopping tablet replica
I20260924 04:41:43.425987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 7252
I20260924 04:41:43.425994 7874 raft_consensus.cc:2258] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 5faf10a6f2b641e886914ca2c67a591c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:43.427048 7873 raft_consensus.cc:2229] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:43.427313 7873 raft_consensus.cc:2258] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:43.428084 7872 raft_consensus.cc:2258] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:43.428433 7873 ts_tablet_manager.cc:1905] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:43.428884 7872 ts_tablet_manager.cc:1905] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:43.431340 7873 ts_tablet_manager.cc:1918] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:43.431475 7872 ts_tablet_manager.cc:1918] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:43.431497 7873 log.cc:1196] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/9e7a1f3a94c142aeafbd1d8abdad57b9
I20260924 04:41:43.431620 7872 log.cc:1196] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/9e7a1f3a94c142aeafbd1d8abdad57b9
I20260924 04:41:43.431824 7873 ts_tablet_manager.cc:1939] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8fbb6f7977324456ab7632d01606d03e: Deleting consensus metadata
I20260924 04:41:43.431969 7872 ts_tablet_manager.cc:1939] T 9e7a1f3a94c142aeafbd1d8abdad57b9 P 8d2f8948fdd44b9590a13d124de15253: Deleting consensus metadata
I20260924 04:41:43.432688 7279 catalog_manager.cc:4841] TS 8d2f8948fdd44b9590a13d124de15253 (127.0.98.2:35289): tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 (table test-table [id=7d4e3fc8f7174ed4885e0b8c666f8ccb]) successfully deleted
I20260924 04:41:43.432688 7277 catalog_manager.cc:4841] TS 8fbb6f7977324456ab7632d01606d03e (127.0.98.3:39157): tablet 9e7a1f3a94c142aeafbd1d8abdad57b9 (table test-table [id=7d4e3fc8f7174ed4885e0b8c666f8ccb]) successfully deleted
I20260924 04:41:43.435235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 7479
I20260924 04:41:43.435441 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:43.435106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34639 (local address 127.0.0.1:48966)
0924 04:41:43.435273 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:43.435340 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 04:41:43.436842 7278 catalog_manager.cc:4584] TS 5faf10a6f2b641e886914ca2c67a591c (127.0.98.1:34639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e7a1f3a94c142aeafbd1d8abdad57b9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34639: connect: Connection refused (error 111)
I20260924 04:41:43.441756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 7615
I20260924 04:41:43.447798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 6971
I20260924 04:41:43.453527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 7079
2026-09-24T04:41:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7175 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:43 dist-test-slave-npg2 krb5kdc[7877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:43 dist-test-slave-npg2 krb5kdc[7877](info): set up 2 sockets
Sep 24 04:41:43 dist-test-slave-npg2 krb5kdc[7877](info): commencing operation
krb5kdc: starting...
W20260924 04:41:44.499797 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:44 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:41:44.627457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:44.745076 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:44.745375 7890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:44.745417 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:44.749509 7890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:44.749585 7890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:41:44.749603 7890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:44.749622 7890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:44.749639 7890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:44.749655 7890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:44.755410 7890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:44.756471 7890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:44.757432 7890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:44.766259 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:44.766248 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:44.766389 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:44.766642 7890 server_base.cc:1034] running on GCE node
I20260924 04:41:44.767395 7890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:44.768543 7890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:44.769740 7890 hybrid_clock.cc:648] HybridClock initialized: now 1790224904769712 us; error 47 us; skew 500 ppm
Sep 24 04:41:44 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:44.772914 7890 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:41:44.773962 7890 webserver.cc:460] Webserver started at http://127.0.98.62:35675/ using document root <none> and password file <none>
I20260924 04:41:44.774590 7890 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:44.774693 7890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:44.774928 7890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:44.777076 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "6e81de2d4e264a85a330a797741ba1b7"
format_stamp: "Formatted at 2026-09-24 04:41:44 on dist-test-slave-npg2"
server_key: "6748572ca2c51457dc2a6a1a1230a93a"
server_key_iv: "46db1c08a50369c1569432f4937ae25a"
server_key_version: "encryptionkey@0"
I20260924 04:41:44.777635 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e81de2d4e264a85a330a797741ba1b7"
format_stamp: "Formatted at 2026-09-24 04:41:44 on dist-test-slave-npg2"
server_key: "6748572ca2c51457dc2a6a1a1230a93a"
server_key_iv: "46db1c08a50369c1569432f4937ae25a"
server_key_version: "encryptionkey@0"
I20260924 04:41:44.781548 7890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:44.784178 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:44.785864 7890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:44.786036 7890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "6e81de2d4e264a85a330a797741ba1b7"
format_stamp: "Formatted at 2026-09-24 04:41:44 on dist-test-slave-npg2"
server_key: "6748572ca2c51457dc2a6a1a1230a93a"
server_key_iv: "46db1c08a50369c1569432f4937ae25a"
server_key_version: "encryptionkey@0"
I20260924 04:41:44.786164 7890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:44.800033 7890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:44.803694 7890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:44.803932 7890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:44.811940 7890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37411
I20260924 04:41:44.811990 7967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37411 every 8 connection(s)
I20260924 04:41:44.812916 7890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:44.814317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 7890
I20260924 04:41:44.814430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:44.814762 392 external_mini_cluster.cc:1428] Setting key 4d627d0688ef3e7df6004030381a8310
I20260924 04:41:44.816458 7968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:44.823678 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Bootstrap starting.
Sep 24 04:41:44 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:44.826565 7968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:44.827962 7968 log.cc:824] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:44.830421 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: No bootstrap required, opened a new log
I20260924 04:41:44.830696 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:44.816346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.0.98.62:37411)
0924 04:41:44.816764 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:44.816783 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:44.816824 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:41:44.817646 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:44.817670 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:44.818097 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:44.818511 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:44.819293 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:44.820819 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:44.821731 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:44.822421 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:44.822596 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:44.825437 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:44.825443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:44.825460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:44.825492 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:44.827594 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:44.828044 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:44.828049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:44.828056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:44.828192 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:44.828497 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:44.828500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:44.828501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:44.828971 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:44.829158 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:44.829424 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 04:41:44.829723 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":148,"threads_started":1}
I20260924 04:41:44.834030 7968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e81de2d4e264a85a330a797741ba1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37411 } }
I20260924 04:41:44.834280 7968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:44.834347 7968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e81de2d4e264a85a330a797741ba1b7, State: Initialized, Role: FOLLOWER
I20260924 04:41:44.834717 7968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e81de2d4e264a85a330a797741ba1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37411 } }
I20260924 04:41:44.834833 7968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:44.834872 7968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:44.834940 7968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:44.836026 7968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e81de2d4e264a85a330a797741ba1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37411 } }
I20260924 04:41:44.836385 7968 leader_election.cc:302] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e81de2d4e264a85a330a797741ba1b7; no voters:
I20260924 04:41:44.836717 7968 leader_election.cc:288] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:44.836956 7973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:44.837190 7973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6e81de2d4e264a85a330a797741ba1b7, State: Running, Role: LEADER
I20260924 04:41:44.837548 7973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e81de2d4e264a85a330a797741ba1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37411 } }
I20260924 04:41:44.838115 7968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:44.839219 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e81de2d4e264a85a330a797741ba1b7. Latest consensus state: current_term: 1 leader_uuid: "6e81de2d4e264a85a330a797741ba1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e81de2d4e264a85a330a797741ba1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37411 } } }
I20260924 04:41:44.839358 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:44.839675 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e81de2d4e264a85a330a797741ba1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e81de2d4e264a85a330a797741ba1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37411 } } }
I20260924 04:41:44.839746 7982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:44.839848 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:44.843255 7982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:44.848616 7982 catalog_manager.cc:1347] Generated new cluster ID: c6717cc7271b466ca8cb6bcd100e4488
I20260924 04:41:44.848711 7982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:44.883742 7982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:44.884747 7982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:44.893374 7982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Generated new TSK 0
I20260924 04:41:44.894114 7982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:41:44.930814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37411
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:45.044566 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:45.044838 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:45.044883 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:45.049052 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:45.049118 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:45.049158 7994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:45.049250 7994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:45.054507 7994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:45.055680 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:45.056763 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:45.066439 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:45.066634 7994 server_base.cc:1034] running on GCE node
W20260924 04:41:45.066445 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:45.066481 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:45.067253 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:45.068080 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:45.069329 7994 hybrid_clock.cc:648] HybridClock initialized: now 1790224905069313 us; error 30 us; skew 500 ppm
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:45.072633 7994 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:41:45.073592 7994 webserver.cc:460] Webserver started at http://127.0.98.1:36683/ using document root <none> and password file <none>
I20260924 04:41:45.074221 7994 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:45.074327 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:45.074599 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:45.076854 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bbeb61d9e7e4ae18f931f809150a866"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "0bc84a0eaff2f84f439fa8f6cd354000"
server_key_iv: "64f608a39e05c331a7040881d872b5f1"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.077463 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bbeb61d9e7e4ae18f931f809150a866"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "0bc84a0eaff2f84f439fa8f6cd354000"
server_key_iv: "64f608a39e05c331a7040881d872b5f1"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.081548 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:45.084203 8009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:45.085575 7994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:45.085816 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "1bbeb61d9e7e4ae18f931f809150a866"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "0bc84a0eaff2f84f439fa8f6cd354000"
server_key_iv: "64f608a39e05c331a7040881d872b5f1"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.085937 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:45.111356 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:45.114979 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:45.115242 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:45.116722 7994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:45.116801 7994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:45.116874 7994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:45.116926 7994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:45.128376 7994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38383
I20260924 04:41:45.128396 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38383 every 8 connection(s)
I20260924 04:41:45.129407 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:45.131707 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.117735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56515 (local address 127.0.98.62:37411)
0924 04:41:45.117873 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.117876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.118963 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 04:41:45.120291 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.120298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.120381 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.120446 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.122374 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.123357 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.124566 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.124879 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.125015 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.127824 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.127828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.127833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.127865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.129524 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.130137 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.130141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.130143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.130244 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.130613 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.130615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.130617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.130797 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.130970 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.131484 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.131592 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:41:45.132642 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.118281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.98.1:56515)
0924 04:41:45.118780 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.118822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.119997 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.120978 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.120984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.121478 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.122206 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.122228 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.123501 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.123505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.124452 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.124465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.125065 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.125069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.125257 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.125886 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.125902 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.127640 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.129630 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.129636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.129651 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.129974 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.130357 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.130360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.130364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.130514 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.130966 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.130971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.131265 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.131431 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.131689 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260924 04:41:45.134075 8125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37411
I20260924 04:41:45.134208 8125 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:45.134722 8125 heartbeater.cc:511] Master 127.0.98.62:37411 requested a full tablet report, sending...
I20260924 04:41:45.136401 7921 ts_manager.cc:194] Registered new tserver with Master: 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383)
I20260924 04:41:45.137714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 7994
I20260924 04:41:45.137854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:45.137917 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56515
I20260924 04:41:45.138163 392 external_mini_cluster.cc:1428] Setting key 21e2602485d8d26569b582dce71f6a2a
I20260924 04:41:45.144598 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.136600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58709 (local address 127.0.98.62:37411)
0924 04:41:45.136719 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.136723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.136879 (+ 156us) server_negotiation.cc:407] Connection header received
0924 04:41:45.136981 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.136983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.137018 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.137079 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.137607 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.138385 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.139783 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.140063 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.140122 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.141461 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.141464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.141466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.141486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.142833 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.143347 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.143350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.143351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.143465 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.143790 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.143792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.143794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.143967 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.144047 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.144346 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.144478 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:41:45.144866 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.136473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.98.1:58709)
0924 04:41:45.136789 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.136804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.136902 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.137125 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.137128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.137341 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.137463 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.137470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.138570 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.138574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.139612 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.139623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.140247 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.140249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.140313 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.140834 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.140857 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.141325 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.142946 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.142952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.142958 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.143253 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.143574 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.143580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.143584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.143694 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.144065 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.144070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.144176 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.144296 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.144474 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:41:45.167315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37411
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:45.280087 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:45.280369 8134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:45.280416 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:45.284858 8134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:45.284930 8134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:45.284971 8134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:45.285037 8134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:45.290799 8134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:45.292083 8134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:45.293201 8134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:45.303359 8142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:45.303453 8140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:45.303486 8139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:45.303910 8134 server_base.cc:1034] running on GCE node
I20260924 04:41:45.304330 8134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:45.305007 8134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:45.306200 8134 hybrid_clock.cc:648] HybridClock initialized: now 1790224905306174 us; error 40 us; skew 500 ppm
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:45.309263 8134 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:41:45.310231 8134 webserver.cc:460] Webserver started at http://127.0.98.2:37709/ using document root <none> and password file <none>
I20260924 04:41:45.310877 8134 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:45.310974 8134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:45.311285 8134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:45.313570 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "e7568984f90e5f83e019614160e95070"
server_key_iv: "75b6d71395b983ca39c0d4cbb289b4ac"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.314173 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "e7568984f90e5f83e019614160e95070"
server_key_iv: "75b6d71395b983ca39c0d4cbb289b4ac"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.318101 8134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 04:41:45.320690 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:45.322005 8134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:41:45.322151 8134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "e7568984f90e5f83e019614160e95070"
server_key_iv: "75b6d71395b983ca39c0d4cbb289b4ac"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.322263 8134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:45.352483 8134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:45.356117 8134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:45.356323 8134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:45.357754 8134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:45.357831 8134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:45.357903 8134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:45.357956 8134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:45.369226 8134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34865
I20260924 04:41:45.369313 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34865 every 8 connection(s)
I20260924 04:41:45.370307 8134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:45.374068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 8134
I20260924 04:41:45.374094 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.358836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53801 (local address 127.0.98.62:37411)
0924 04:41:45.358973 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.358977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.360038 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 04:41:45.361234 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.361239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.361319 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.361440 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.363170 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.364097 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.365941 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.366307 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.366448 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.370025 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.370028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.370031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.370060 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.372031 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.372525 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.372528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.372529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.372612 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.372887 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.372889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.372890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.373082 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.373166 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.373692 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.373897 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:41:45.374208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:45.374548 392 external_mini_cluster.cc:1428] Setting key cd7ca3aed32475a9ca334b6b4ac37a5a
I20260924 04:41:45.374807 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.359367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.98.2:53801)
0924 04:41:45.359866 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.359908 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.360906 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.361663 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.361674 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.362173 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.362947 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.362969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.364295 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.364299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.365811 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.365821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.366512 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.366515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.366680 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.367682 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.367708 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.369775 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.372150 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.372155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.372166 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.372431 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.372692 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.372694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.372695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.372820 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.373172 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.373177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.373429 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.373587 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.373842 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":140,"threads_started":1}
I20260924 04:41:45.376461 8265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37411
I20260924 04:41:45.376616 8265 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:45.377935 8265 heartbeater.cc:511] Master 127.0.98.62:37411 requested a full tablet report, sending...
I20260924 04:41:45.379000 7921 ts_manager.cc:194] Registered new tserver with Master: 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865)
I20260924 04:41:45.379848 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53801
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:41:45.386482 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.378617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59723 (local address 127.0.98.62:37411)
0924 04:41:45.378781 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.378786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.379307 (+ 521us) server_negotiation.cc:407] Connection header received
0924 04:41:45.379421 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.379427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.379487 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.379579 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.380076 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.380740 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.381724 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.381960 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.382070 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.383207 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.383210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.383213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.383236 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.384781 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.385252 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.385257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.385258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.385340 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.385636 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.385639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.385641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.385842 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.385958 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.386211 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.386320 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:41:45.386492 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.378895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.98.2:59723)
0924 04:41:45.379222 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.379237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.379328 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.379608 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.379611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.379819 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.379943 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.379950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.380909 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.380913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.381602 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.381611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.382073 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.382077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.382165 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.382714 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.382726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.383047 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.384948 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.384952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.384954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.385152 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.385429 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.385434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.385435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.385523 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.385945 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.385948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.386045 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.386150 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.386290 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":60,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:41:45.406358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37411
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:45.525190 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:45.525533 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:45.525622 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:45.529894 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:45.530005 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:45.530083 8274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:41:45.530164 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:45.535956 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:45.537106 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:45.538203 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:45.548810 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:45.548810 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:45.548869 8274 server_base.cc:1034] running on GCE node
W20260924 04:41:45.549125 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:45.549532 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:45.550242 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:45.551432 8274 hybrid_clock.cc:648] HybridClock initialized: now 1790224905551391 us; error 54 us; skew 500 ppm
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:45.555507 8274 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:41:45.556632 8274 webserver.cc:460] Webserver started at http://127.0.98.3:37683/ using document root <none> and password file <none>
I20260924 04:41:45.557240 8274 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:45.557305 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:45.557562 8274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:45.559859 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "cca80a60690d496b9c442bb4f4affd68"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "7dd7cd9c988b6e21361d3762161f8533"
server_key_iv: "494409de66adf0a6123b7c8ffe29f712"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.560456 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cca80a60690d496b9c442bb4f4affd68"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "7dd7cd9c988b6e21361d3762161f8533"
server_key_iv: "494409de66adf0a6123b7c8ffe29f712"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.564285 8274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:45.566814 8289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:45.568181 8274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:45.568326 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "cca80a60690d496b9c442bb4f4affd68"
format_stamp: "Formatted at 2026-09-24 04:41:45 on dist-test-slave-npg2"
server_key: "7dd7cd9c988b6e21361d3762161f8533"
server_key_iv: "494409de66adf0a6123b7c8ffe29f712"
server_key_version: "encryptionkey@0"
I20260924 04:41:45.568485 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:45.582896 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:45.586253 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:45.586486 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:45.588177 8274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:45.588263 8274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:45.588356 8274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:45.588410 8274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:45.599530 8274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34157
I20260924 04:41:45.599568 8404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34157 every 8 connection(s)
I20260924 04:41:45.600662 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:45.602326 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.588882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54091 (local address 127.0.98.62:37411)
0924 04:41:45.589026 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.589031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.590098 (+ 1067us) server_negotiation.cc:407] Connection header received
0924 04:41:45.591190 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.591194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.591253 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.591334 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.592711 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.593330 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.594618 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.594933 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.595060 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.598352 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.598356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.598359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.598388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.600095 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.600688 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.600693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.600698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.600790 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.601116 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.601121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.601123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.601366 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.601490 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.602016 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.602150 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:41:45.603180 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.589371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.98.3:54091)
0924 04:41:45.589850 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.590004 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.590889 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.591521 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.591532 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.592004 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.592569 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.592583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.593460 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.593463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.594495 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.594504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.595153 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.595165 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.595395 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.596067 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.596093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.598141 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.600284 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.600290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.600303 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.600573 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.600879 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.600881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.600882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.601012 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.601485 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.601492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.601732 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.601974 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.602290 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260924 04:41:45.603272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 8274
I20260924 04:41:45.603416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:45.603701 392 external_mini_cluster.cc:1428] Setting key 57fde7b6b2a1440b1c371d483c35af19
I20260924 04:41:45.604454 8405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37411
I20260924 04:41:45.604627 8405 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:45.605209 8405 heartbeater.cc:511] Master 127.0.98.62:37411 requested a full tablet report, sending...
I20260924 04:41:45.606331 7920 ts_manager.cc:194] Registered new tserver with Master: cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:45.607144 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54091
I20260924 04:41:45.613543 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.606665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54951 (local address 127.0.98.62:37411)
0924 04:41:45.606788 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.606792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.606975 (+ 183us) server_negotiation.cc:407] Connection header received
0924 04:41:45.607066 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.607068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.607103 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.607242 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.607784 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.608550 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.609370 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.609568 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.609648 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.610641 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.610644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.610646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.610668 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.612147 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.612582 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.612585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.612586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.612654 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.612900 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.612902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.612904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.613052 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.613127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.613335 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.613439 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:41:45.613547 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.606605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.98.3:54951)
0924 04:41:45.606887 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.606902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.606991 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.607254 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.607257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.607475 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.607598 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.607617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.608694 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.608697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.609256 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.609264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.609664 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.609667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.609733 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.610170 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.610181 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.610525 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.612294 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.612297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.612299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.612493 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.612742 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.612744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.612746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.612821 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.613130 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.613132 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.613205 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.613303 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.613408 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
I20260924 04:41:45.617516 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:41:45.641975 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:45.635441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.0.98.62:37411)
0924 04:41:45.635571 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.635575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.635587 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:45.635627 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.635629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.635666 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.635760 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.636511 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.637032 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.638002 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.638218 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.638330 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.641823 (+ 3493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54306: BlockingRecv error: recv got EOF from 127.0.0.1:54306 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:45.651678 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.643464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.0.98.62:37411)
0924 04:41:45.643581 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.643584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.643746 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:41:45.643850 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.643852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.643895 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.643964 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.644571 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.645049 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.645840 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.646075 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.646169 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.648546 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.648550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.648552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.648574 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.650039 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.650472 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.650483 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.650485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.650557 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.650829 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.650834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.650835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.651012 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.651169 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.651397 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.651512 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:41:45.654572 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:45.657689 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:45.672063 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.665954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.0.1:45372)
0924 04:41:45.666328 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.666382 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.666518 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.667388 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.667394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.667417 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:45.667550 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.667556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.669102 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.669105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.670413 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.670422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.671510 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.671514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.671618 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.671669 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.671773 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.671909 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260924 04:41:45.672205 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.666733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.0.98.3:34157)
0924 04:41:45.667205 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.667217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.667238 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:41:45.667310 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.667314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.667535 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.667947 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:45.668072 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.669125 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.670800 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.671625 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.671727 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.671929 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.671978 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.672061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":110,"threads_started":1}
I20260924 04:41:45.672354 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.666674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.0.1:59010)
0924 04:41:45.667625 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.667637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.667724 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.668134 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.668136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.668153 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:45.668263 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.668269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.669363 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.669367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.670674 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.670681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.671918 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.671922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.672016 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.672035 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.672154 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.672237 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":110,"thread_start_us":86,"threads_started":1}
I20260924 04:41:45.672137 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.666193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34157 (local address 127.0.0.1:43810)
0924 04:41:45.666723 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.666737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.666815 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.667683 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.667686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.667706 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:45.667789 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.667794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.669238 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.669243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.670671 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.670681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.671738 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.671741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.671827 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.671849 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.671914 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.671988 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":112,"thread_start_us":93,"threads_started":1}
I20260924 04:41:45.672762 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.666087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.0.98.2:34865)
0924 04:41:45.666832 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.666846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.666872 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:41:45.666952 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.666958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.667221 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.667385 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:45.667693 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.668930 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.670537 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.671383 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.671499 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.671663 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.672494 (+ 831us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.672602 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":111,"threads_started":1}
I20260924 04:41:45.673044 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.667101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.0.98.1:38383)
0924 04:41:45.667658 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.667667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.667692 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:41:45.667875 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.667879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.668007 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.668145 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:45.668382 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.669235 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.670814 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.671617 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.671716 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.671858 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.672789 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.672902 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":89,"threads_started":1}
I20260924 04:41:45.675654 8199 tablet_service.cc:1467] Processing CreateTablet for tablet e0473ce07d774425bb979506db19e80b (DEFAULT_TABLE table=test-table [id=9455e01b45644bc6956b8bb1db02ce28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:45.675745 8059 tablet_service.cc:1467] Processing CreateTablet for tablet e0473ce07d774425bb979506db19e80b (DEFAULT_TABLE table=test-table [id=9455e01b45644bc6956b8bb1db02ce28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:45.675746 8339 tablet_service.cc:1467] Processing CreateTablet for tablet e0473ce07d774425bb979506db19e80b (DEFAULT_TABLE table=test-table [id=9455e01b45644bc6956b8bb1db02ce28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:45.676687 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0473ce07d774425bb979506db19e80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.676734 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0473ce07d774425bb979506db19e80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.676738 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0473ce07d774425bb979506db19e80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.683449 8433 tablet_bootstrap.cc:492] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Bootstrap starting.
I20260924 04:41:45.684453 8434 tablet_bootstrap.cc:492] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Bootstrap starting.
I20260924 04:41:45.685540 8433 tablet_bootstrap.cc:654] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.686146 8434 tablet_bootstrap.cc:654] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.686290 8433 log.cc:824] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:45.686838 8434 log.cc:824] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:45.687003 8435 tablet_bootstrap.cc:492] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Bootstrap starting.
I20260924 04:41:45.688441 8433 tablet_bootstrap.cc:492] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: No bootstrap required, opened a new log
I20260924 04:41:45.688648 8433 ts_tablet_manager.cc:1397] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:41:45.688709 8434 tablet_bootstrap.cc:492] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: No bootstrap required, opened a new log
I20260924 04:41:45.688934 8434 ts_tablet_manager.cc:1397] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260924 04:41:45.689311 8435 tablet_bootstrap.cc:654] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.690129 8435 log.cc:824] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:45.692323 8435 tablet_bootstrap.cc:492] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: No bootstrap required, opened a new log
I20260924 04:41:45.692517 8435 ts_tablet_manager.cc:1397] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:41:45.692476 8433 raft_consensus.cc:348] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.692764 8433 raft_consensus.cc:374] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.692847 8433 raft_consensus.cc:729] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45465cc3524a68ad2f4b011ea1a5dd, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.693291 8433 consensus_queue.cc:260] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.693501 8434 raft_consensus.cc:348] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.693802 8434 raft_consensus.cc:374] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.693876 8434 raft_consensus.cc:729] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.693941 8265 heartbeater.cc:503] Master 127.0.98.62:37411 was elected leader, sending a full tablet report...
I20260924 04:41:45.694283 8434 consensus_queue.cc:260] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.695364 8434 ts_tablet_manager.cc:1428] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 04:41:45.695477 8433 ts_tablet_manager.cc:1428] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260924 04:41:45.695724 8125 heartbeater.cc:503] Master 127.0.98.62:37411 was elected leader, sending a full tablet report...
I20260924 04:41:45.696447 8435 raft_consensus.cc:348] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.696689 8435 raft_consensus.cc:374] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.696741 8435 raft_consensus.cc:729] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.697150 8435 consensus_queue.cc:260] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.697744 8405 heartbeater.cc:503] Master 127.0.98.62:37411 was elected leader, sending a full tablet report...
I20260924 04:41:45.699097 8435 ts_tablet_manager.cc:1428] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:41:45.751499 8441 raft_consensus.cc:482] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:45.751696 8441 raft_consensus.cc:504] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.752821 8441 leader_election.cc:288] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865), 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383)
I20260924 04:41:45.756722 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.753150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.98.3:59771)
0924 04:41:45.753293 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.753310 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.753414 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.753714 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.753717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.753737 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:45.753920 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.753927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.754677 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.754684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.755579 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.755588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.756272 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.756275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.756387 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.756430 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.756509 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.756581 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260924 04:41:45.756722 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.753277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59771 (local address 127.0.98.2:34865)
0924 04:41:45.753434 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.753439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.753460 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:41:45.753520 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.753525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.753581 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.753677 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:45.754048 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.754569 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.755699 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.756114 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.756264 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.756340 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.756548 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.756620 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:41:45.757372 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0473ce07d774425bb979506db19e80b" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" is_pre_election: true
I20260924 04:41:45.757675 8219 raft_consensus.cc:2454] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 0.
I20260924 04:41:45.757712 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.753583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58497 (local address 127.0.98.1:38383)
0924 04:41:45.753829 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.753834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.754014 (+ 180us) server_negotiation.cc:407] Connection header received
0924 04:41:45.754141 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.754145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.754197 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.754291 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:45.754953 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.755519 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.756567 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.757156 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.757289 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.757345 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.757523 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.757595 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 04:41:45.757716 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.753442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.98.3:58497)
0924 04:41:45.753919 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.753930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.754050 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.754302 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.754304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.754321 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:45.754824 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.754831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.755628 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.755633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.756434 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.756444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.757297 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.757300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.757393 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.757421 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.757509 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.757573 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":172,"threads_started":1}
I20260924 04:41:45.758210 8292 leader_election.cc:302] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c45465cc3524a68ad2f4b011ea1a5dd, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:45.758376 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0473ce07d774425bb979506db19e80b" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbeb61d9e7e4ae18f931f809150a866" is_pre_election: true
I20260924 04:41:45.758522 8441 raft_consensus.cc:2781] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:45.758626 8441 raft_consensus.cc:482] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:45.758666 8079 raft_consensus.cc:2454] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 0.
I20260924 04:41:45.758685 8441 raft_consensus.cc:3037] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:45.759974 8441 raft_consensus.cc:504] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.760313 8441 leader_election.cc:288] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865), 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383)
I20260924 04:41:45.760649 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0473ce07d774425bb979506db19e80b" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
I20260924 04:41:45.760763 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0473ce07d774425bb979506db19e80b" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbeb61d9e7e4ae18f931f809150a866"
I20260924 04:41:45.760782 8219 raft_consensus.cc:3037] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:45.760885 8079 raft_consensus.cc:3037] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:45.762164 8079 raft_consensus.cc:2454] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 1.
I20260924 04:41:45.762480 8292 leader_election.cc:302] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:45.762538 8219 raft_consensus.cc:2454] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 1.
I20260924 04:41:45.762668 8441 raft_consensus.cc:2781] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:45.762931 8441 raft_consensus.cc:686] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 1 LEADER]: Becoming Leader. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Running, Role: LEADER
I20260924 04:41:45.763356 8441 consensus_queue.cc:237] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.766728 7921 catalog_manager.cc:5495] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca80a60690d496b9c442bb4f4affd68 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cca80a60690d496b9c442bb4f4affd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } health_report { overall_health: HEALTHY } } }
I20260924 04:41:45.797364 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.793330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.98.3:34157)
0924 04:41:45.793475 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.793479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.793494 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:41:45.793646 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.793649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.793701 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.793921 (+ 220us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:45.794164 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.794713 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.795864 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.796092 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.796202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.796504 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:45.796963 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:45.797091 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.797154 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.797212 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:41:45.803674 8219 raft_consensus.cc:1261] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Refusing update from remote peer cca80a60690d496b9c442bb4f4affd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:45.803674 8079 raft_consensus.cc:1261] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Refusing update from remote peer cca80a60690d496b9c442bb4f4affd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:45.804467 8443 consensus_queue.cc:1035] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:45.804690 8441 consensus_queue.cc:1035] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:45.813548 8448 mvcc.cc:205] Tried to move back new op lower bound from 7332761214163734528 to 7332761214009589760. Current Snapshot: MvccSnapshot[applied={T|T < 7332761214163734528}]
I20260924 04:41:45.814661 8447 mvcc.cc:205] Tried to move back new op lower bound from 7332761214163734528 to 7332761214009589760. Current Snapshot: MvccSnapshot[applied={T|T < 7332761214163734528}]
I20260924 04:41:45.819221 8446 mvcc.cc:205] Tried to move back new op lower bound from 7332761214163734528 to 7332761214009589760. Current Snapshot: MvccSnapshot[applied={T|T < 7332761214163734528 or (T in {7332761214163734528})}]
I20260924 04:41:45.823908 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:45.824090 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:45.826741 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Sending DeleteTablet for 3 replicas of tablet e0473ce07d774425bb979506db19e80b
I20260924 04:41:45.827698 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet e0473ce07d774425bb979506db19e80b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:45 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43810
I20260924 04:41:45.827708 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet e0473ce07d774425bb979506db19e80b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:45 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59010
I20260924 04:41:45.827836 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet e0473ce07d774425bb979506db19e80b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:45 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45372
I20260924 04:41:45.828385 8456 tablet_replica.cc:330] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: stopping tablet replica
I20260924 04:41:45.828723 8456 raft_consensus.cc:2229] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:45.828995 8455 tablet_replica.cc:330] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: stopping tablet replica
I20260924 04:41:45.829067 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:41:45.829243 8456 raft_consensus.cc:2258] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:45.829330 8455 raft_consensus.cc:2229] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:41:45.829550 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:45.829635 8455 raft_consensus.cc:2258] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:45.831039 8455 ts_tablet_manager.cc:1905] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:45.831039 8456 ts_tablet_manager.cc:1905] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:45.831308 8457 tablet_replica.cc:330] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: stopping tablet replica
I20260924 04:41:45.831611 8457 raft_consensus.cc:2229] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:45.831907 8457 raft_consensus.cc:2258] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:45.834100 8455 ts_tablet_manager.cc:1918] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:45.834380 8455 log.cc:1196] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/e0473ce07d774425bb979506db19e80b
I20260924 04:41:45.834812 8455 ts_tablet_manager.cc:1939] T e0473ce07d774425bb979506db19e80b P 1bbeb61d9e7e4ae18f931f809150a866: Deleting consensus metadata
I20260924 04:41:45.835486 8457 ts_tablet_manager.cc:1905] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:45.835846 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 188e16b478ca49a58d7577516c6d3c3e (DEFAULT_TABLE table=test-table [id=f7c6b8e4e215483fa76aa4b2b755cfab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:45.836095 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e16b478ca49a58d7577516c6d3c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.836804 8456 ts_tablet_manager.cc:1918] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:45.836889 8456 log.cc:1196] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/e0473ce07d774425bb979506db19e80b
I20260924 04:41:45.837142 8456 ts_tablet_manager.cc:1939] T e0473ce07d774425bb979506db19e80b P cca80a60690d496b9c442bb4f4affd68: Deleting consensus metadata
I20260924 04:41:45.838153 7906 catalog_manager.cc:4841] TS cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157): tablet e0473ce07d774425bb979506db19e80b (table test-table [id=9455e01b45644bc6956b8bb1db02ce28]) successfully deleted
I20260924 04:41:45.838433 8435 tablet_bootstrap.cc:492] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Bootstrap starting.
I20260924 04:41:45.838611 7908 catalog_manager.cc:4841] TS 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383): tablet e0473ce07d774425bb979506db19e80b (table test-table [id=9455e01b45644bc6956b8bb1db02ce28]) successfully deleted
I20260924 04:41:45.839135 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 188e16b478ca49a58d7577516c6d3c3e (DEFAULT_TABLE table=test-table [id=f7c6b8e4e215483fa76aa4b2b755cfab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:45.839385 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e16b478ca49a58d7577516c6d3c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.839533 8435 tablet_bootstrap.cc:654] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.840535 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 188e16b478ca49a58d7577516c6d3c3e (DEFAULT_TABLE table=test-table [id=f7c6b8e4e215483fa76aa4b2b755cfab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:45.840788 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e16b478ca49a58d7577516c6d3c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.841073 8457 ts_tablet_manager.cc:1918] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:45.841164 8457 log.cc:1196] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/e0473ce07d774425bb979506db19e80b
I20260924 04:41:45.841491 8457 ts_tablet_manager.cc:1939] T e0473ce07d774425bb979506db19e80b P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting consensus metadata
I20260924 04:41:45.841616 8433 tablet_bootstrap.cc:492] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Bootstrap starting.
I20260924 04:41:45.842489 8433 tablet_bootstrap.cc:654] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.843364 8434 tablet_bootstrap.cc:492] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Bootstrap starting.
I20260924 04:41:45.843381 7908 catalog_manager.cc:4841] TS 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865): tablet e0473ce07d774425bb979506db19e80b (table test-table [id=9455e01b45644bc6956b8bb1db02ce28]) successfully deleted
I20260924 04:41:45.844419 8434 tablet_bootstrap.cc:654] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.844902 8435 tablet_bootstrap.cc:492] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: No bootstrap required, opened a new log
I20260924 04:41:45.844990 8435 ts_tablet_manager.cc:1397] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 04:41:45.845494 8435 raft_consensus.cc:348] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.845603 8435 raft_consensus.cc:374] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.845600 8434 tablet_bootstrap.cc:492] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: No bootstrap required, opened a new log
I20260924 04:41:45.845624 8433 tablet_bootstrap.cc:492] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: No bootstrap required, opened a new log
I20260924 04:41:45.845633 8435 raft_consensus.cc:729] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.845685 8434 ts_tablet_manager.cc:1397] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:45.845718 8433 ts_tablet_manager.cc:1397] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:41:45.845759 8435 consensus_queue.cc:260] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.846002 8435 ts_tablet_manager.cc:1428] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:45.846280 8433 raft_consensus.cc:348] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.846403 8433 raft_consensus.cc:374] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.846443 8433 raft_consensus.cc:729] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45465cc3524a68ad2f4b011ea1a5dd, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.846575 8433 consensus_queue.cc:260] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.846621 8434 raft_consensus.cc:348] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.846726 8434 raft_consensus.cc:374] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.846771 8434 raft_consensus.cc:729] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.846939 8433 ts_tablet_manager.cc:1428] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:41:45.847039 8434 consensus_queue.cc:260] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.848055 8434 ts_tablet_manager.cc:1428] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 04:41:45.851943 8406 tablet.cc:2369] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:45.871477 8266 tablet.cc:2369] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:41:45.880848 8126 tablet.cc:2369] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:41:45.931740 8441 raft_consensus.cc:482] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:45.931911 8441 raft_consensus.cc:504] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.932349 8441 leader_election.cc:288] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383), 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865)
I20260924 04:41:45.932950 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e16b478ca49a58d7577516c6d3c3e" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" is_pre_election: true
I20260924 04:41:45.932940 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e16b478ca49a58d7577516c6d3c3e" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbeb61d9e7e4ae18f931f809150a866" is_pre_election: true
I20260924 04:41:45.933102 8219 raft_consensus.cc:2454] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 0.
I20260924 04:41:45.933115 8079 raft_consensus.cc:2454] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 0.
I20260924 04:41:45.933485 8292 leader_election.cc:302] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:45.933665 8441 raft_consensus.cc:2781] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:45.933722 8441 raft_consensus.cc:482] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:45.933769 8441 raft_consensus.cc:3037] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:45.934964 8441 raft_consensus.cc:504] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.935380 8441 leader_election.cc:288] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 election: Requested vote from peers 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383), 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865)
I20260924 04:41:45.935791 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e16b478ca49a58d7577516c6d3c3e" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbeb61d9e7e4ae18f931f809150a866"
I20260924 04:41:45.935798 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e16b478ca49a58d7577516c6d3c3e" candidate_uuid: "cca80a60690d496b9c442bb4f4affd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
I20260924 04:41:45.935922 8219 raft_consensus.cc:3037] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:45.935926 8079 raft_consensus.cc:3037] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:45.936918 8079 raft_consensus.cc:2454] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 1.
I20260924 04:41:45.937062 8219 raft_consensus.cc:2454] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca80a60690d496b9c442bb4f4affd68 in term 1.
I20260924 04:41:45.937269 8292 leader_election.cc:302] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:45.937434 8441 raft_consensus.cc:2781] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:45.937515 8441 raft_consensus.cc:686] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 1 LEADER]: Becoming Leader. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Running, Role: LEADER
I20260924 04:41:45.937656 8441 consensus_queue.cc:237] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.939544 7919 catalog_manager.cc:5495] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca80a60690d496b9c442bb4f4affd68 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cca80a60690d496b9c442bb4f4affd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } health_report { overall_health: HEALTHY } } }
Sep 24 04:41:45 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:45.978967 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.968870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37411 (local address 127.0.0.1:54326)
0924 04:41:45.969166 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:45.969190 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:45.969290 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:45.969590 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:45.969594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:45.970014 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:45.970119 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.970125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.971206 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.971208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.971848 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:45.971857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.972515 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.972520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:45.972609 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.973528 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:45.973540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:45.975078 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:45.977225 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.977236 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.977240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.977528 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.977856 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:45.977859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:45.977861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:45.977948 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:45.978397 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:45.978405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:45.978520 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 04:41:45.978614 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:41:45.978831 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
I20260924 04:41:45.979149 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:45.968976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.0.98.62:37411)
0924 04:41:45.969113 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:45.969118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:45.969266 (+ 148us) server_negotiation.cc:407] Connection header received
0924 04:41:45.969376 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:45.969379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:45.969426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:45.969545 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:45.970244 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.971039 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.971983 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:45.972323 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:45.972425 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:45.975269 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:45.975273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:45.975276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:45.975301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:45.977075 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.977634 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.977637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.977639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.977725 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:45.978039 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:45.978043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:45.978045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:45.978276 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:45.978374 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:45.978722 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:41:45.978926 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":52}
I20260924 04:41:45.983518 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54326:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:41:45.984428 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:45.989490 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 95a975d2ec3b4e4f9537261ec3d86a60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad02d1ce5329411f9709b5d72ba6c86b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:45.989519 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 95a975d2ec3b4e4f9537261ec3d86a60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad02d1ce5329411f9709b5d72ba6c86b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:45.989591 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 95a975d2ec3b4e4f9537261ec3d86a60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad02d1ce5329411f9709b5d72ba6c86b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:41:45.989899 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a975d2ec3b4e4f9537261ec3d86a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.989899 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a975d2ec3b4e4f9537261ec3d86a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.989938 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a975d2ec3b4e4f9537261ec3d86a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:45.992944 8435 tablet_bootstrap.cc:492] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: Bootstrap starting.
I20260924 04:41:45.992975 8434 tablet_bootstrap.cc:492] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: Bootstrap starting.
I20260924 04:41:45.993772 8434 tablet_bootstrap.cc:654] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.993803 8435 tablet_bootstrap.cc:654] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.994663 8433 tablet_bootstrap.cc:492] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: Bootstrap starting.
I20260924 04:41:45.995025 8434 tablet_bootstrap.cc:492] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: No bootstrap required, opened a new log
I20260924 04:41:45.995146 8434 ts_tablet_manager.cc:1397] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:45.995373 8435 tablet_bootstrap.cc:492] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: No bootstrap required, opened a new log
I20260924 04:41:45.995440 8435 ts_tablet_manager.cc:1397] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:45.995671 8433 tablet_bootstrap.cc:654] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:45.995633 8434 raft_consensus.cc:348] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.995759 8434 raft_consensus.cc:374] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.995797 8434 raft_consensus.cc:729] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.995774 8435 raft_consensus.cc:348] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.995874 8435 raft_consensus.cc:374] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.995903 8435 raft_consensus.cc:729] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.995927 8434 consensus_queue.cc:260] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.996014 8435 consensus_queue.cc:260] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.996248 8435 ts_tablet_manager.cc:1428] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:41:45.996253 8434 ts_tablet_manager.cc:1428] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:45.996387 8443 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:45.996517 8443 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:45.996932 8440 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:45.997057 8440 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:45.998132 8433 tablet_bootstrap.cc:492] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: No bootstrap required, opened a new log
I20260924 04:41:45.998246 8433 ts_tablet_manager.cc:1397] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:41:45.998719 8433 raft_consensus.cc:348] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.998854 8433 raft_consensus.cc:374] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:45.998900 8433 raft_consensus.cc:729] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45465cc3524a68ad2f4b011ea1a5dd, State: Initialized, Role: FOLLOWER
I20260924 04:41:45.999035 8433 consensus_queue.cc:260] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:45.999310 8433 ts_tablet_manager.cc:1428] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:45.999280 8439 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:45.999433 8439 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:46.102063 8440 raft_consensus.cc:482] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:46.102200 8440 raft_consensus.cc:504] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.103518 8440 leader_election.cc:288] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865), cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:46.107937 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.103767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34157 (local address 127.0.98.1:41167)
0924 04:41:46.104162 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.104173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.104334 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.104595 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.104598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.104614 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:46.104738 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.104744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.105502 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.105506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.106591 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.106601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.107411 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.107415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.107548 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.107576 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.107654 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.107729 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260924 04:41:46.107949 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.103854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41167 (local address 127.0.98.3:34157)
0924 04:41:46.104016 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.104021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.104269 (+ 248us) server_negotiation.cc:407] Connection header received
0924 04:41:46.104442 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.104450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.104495 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.104574 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:46.104842 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.105337 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.106721 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.107248 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.107330 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.107388 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.107684 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.107760 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 04:41:46.108057 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.103842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53879 (local address 127.0.98.2:34865)
0924 04:41:46.104005 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.104012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.104486 (+ 474us) server_negotiation.cc:407] Connection header received
0924 04:41:46.104643 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.104646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.104698 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.104789 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:46.105089 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.105811 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.106911 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.107463 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.107575 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.107652 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.107837 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.107909 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:41:46.107993 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.103765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.98.1:53879)
0924 04:41:46.104372 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.104389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.104529 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.104800 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.104802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.104819 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:46.104965 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.104971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.105944 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.105946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.106777 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.106787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.107583 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.107586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.107697 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.107722 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.107796 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.107858 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":175,"thread_start_us":150,"threads_started":1}
I20260924 04:41:46.108753 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a975d2ec3b4e4f9537261ec3d86a60" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" is_pre_election: true
I20260924 04:41:46.108738 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a975d2ec3b4e4f9537261ec3d86a60" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca80a60690d496b9c442bb4f4affd68" is_pre_election: true
I20260924 04:41:46.108927 8219 raft_consensus.cc:2454] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 0.
I20260924 04:41:46.109014 8359 raft_consensus.cc:2454] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 0.
I20260924 04:41:46.109299 8012 leader_election.cc:302] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, 9c45465cc3524a68ad2f4b011ea1a5dd; no voters:
I20260924 04:41:46.109629 8440 raft_consensus.cc:2781] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:46.109689 8440 raft_consensus.cc:482] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:46.109727 8440 raft_consensus.cc:3037] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.110697 8440 raft_consensus.cc:504] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.111049 8440 leader_election.cc:288] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865), cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:46.111394 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a975d2ec3b4e4f9537261ec3d86a60" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
I20260924 04:41:46.111442 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a975d2ec3b4e4f9537261ec3d86a60" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca80a60690d496b9c442bb4f4affd68"
I20260924 04:41:46.111536 8219 raft_consensus.cc:3037] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.111552 8359 raft_consensus.cc:3037] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.112874 8359 raft_consensus.cc:2454] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 1.
I20260924 04:41:46.112875 8219 raft_consensus.cc:2454] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 1.
I20260924 04:41:46.113250 8010 leader_election.cc:302] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:46.113510 8440 raft_consensus.cc:2781] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:46.113806 8440 raft_consensus.cc:686] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 LEADER]: Becoming Leader. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Running, Role: LEADER
I20260924 04:41:46.114081 8440 consensus_queue.cc:237] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.115386 8473 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: TxnStatusTablet state changed. Reason: New leader 1bbeb61d9e7e4ae18f931f809150a866. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.115523 8473 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:46.115975 8475 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:41:46.116123 8475 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:41:46.116804 7919 catalog_manager.cc:5495] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:46.159533 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.155045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.0.1:59024)
0924 04:41:46.155327 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.155342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.155447 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.155815 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.155818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.155828 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:46.155949 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.155954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.156964 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.156967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.157692 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.157702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.158271 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.158274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.158367 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.158413 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.159243 (+ 830us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.159256 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.159345 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.159417 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 04:41:46.159569 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.155224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.0.98.1:38383)
0924 04:41:46.155366 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.155371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.155395 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:41:46.155591 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.155594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.155665 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.155793 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.156055 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.156834 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.157809 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.158159 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.158261 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.158554 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.159109 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.159249 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.159374 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.159436 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:41:46.162673 8219 raft_consensus.cc:1261] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Refusing update from remote peer 1bbeb61d9e7e4ae18f931f809150a866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.162815 8359 raft_consensus.cc:1261] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Refusing update from remote peer 1bbeb61d9e7e4ae18f931f809150a866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.163206 8473 consensus_queue.cc:1035] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.163414 8440 consensus_queue.cc:1035] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.164731 8439 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: TxnStatusTablet state changed. Reason: New leader 1bbeb61d9e7e4ae18f931f809150a866. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.164844 8439 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:46.164968 8443 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: TxnStatusTablet state changed. Reason: New leader 1bbeb61d9e7e4ae18f931f809150a866. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.165077 8443 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:46.166873 8440 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.166978 8440 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:46.167104 8440 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.167207 8440 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 1bbeb61d9e7e4ae18f931f809150a866: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:41:46.168751 8439 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.168857 8439 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P 9c45465cc3524a68ad2f4b011ea1a5dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:46.169041 8443 tablet_replica.cc:439] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } } }
I20260924 04:41:46.169159 8443 tablet_replica.cc:442] T 95a975d2ec3b4e4f9537261ec3d86a60 P cca80a60690d496b9c442bb4f4affd68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:41:46.183557 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.179872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38833 (local address 127.0.98.1:38383)
0924 04:41:46.180004 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.180008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.180166 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:41:46.180259 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.180261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.180314 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.180393 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.180646 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.181213 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.182446 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.182658 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.182758 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.183053 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.183185 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.183257 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.183402 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.183454 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:41:46.183604 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.179648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.98.3:38833)
0924 04:41:46.180047 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.180063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.180174 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.180423 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.180426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.180434 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:46.180543 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.180549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.181321 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.181325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.182321 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.182331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.182776 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.182780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.182866 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.182923 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.183281 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.183287 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.183357 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.183443 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":70,"threads_started":1}
I20260924 04:41:46.190987 8219 raft_consensus.cc:1261] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Refusing update from remote peer cca80a60690d496b9c442bb4f4affd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.191077 8079 raft_consensus.cc:1261] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Refusing update from remote peer cca80a60690d496b9c442bb4f4affd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.191404 8443 consensus_queue.cc:1035] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.191552 8441 consensus_queue.cc:1035] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.213421 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.209472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60389 (local address 127.0.98.3:34157)
0924 04:41:46.209573 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.209578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.209890 (+ 312us) server_negotiation.cc:407] Connection header received
0924 04:41:46.209939 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.209942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.209992 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.210071 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.210466 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.211355 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.212203 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.212528 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.212631 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.212927 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.213008 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.213081 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.213227 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.213280 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:41:46.213493 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.209396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34157 (local address 127.0.98.1:60389)
0924 04:41:46.209707 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.209719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.209838 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.210209 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.210212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.210221 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:46.210339 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.210345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.211457 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.211460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.212087 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.212096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.212660 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.212664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.212737 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.212785 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.213123 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.213133 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.213209 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.213355 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":120,"threads_started":1}
I20260924 04:41:46.233722 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:46.233940 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:46.236455 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Sending DeleteTablet for 3 replicas of tablet 188e16b478ca49a58d7577516c6d3c3e
I20260924 04:41:46.237160 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e16b478ca49a58d7577516c6d3c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43810
I20260924 04:41:46.237179 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e16b478ca49a58d7577516c6d3c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45372
I20260924 04:41:46.237183 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e16b478ca49a58d7577516c6d3c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59010
I20260924 04:41:46.237398 8456 tablet_replica.cc:330] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: stopping tablet replica
I20260924 04:41:46.237445 8455 tablet_replica.cc:330] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: stopping tablet replica
I20260924 04:41:46.237540 8455 raft_consensus.cc:2229] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:46.237545 8456 raft_consensus.cc:2229] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:46.237661 8455 raft_consensus.cc:2258] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.237740 8456 raft_consensus.cc:2258] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.237933 8457 tablet_replica.cc:330] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: stopping tablet replica
I20260924 04:41:46.238075 8457 raft_consensus.cc:2229] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:46.238240 8457 raft_consensus.cc:2258] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.238814 8456 ts_tablet_manager.cc:1905] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.238866 8455 ts_tablet_manager.cc:1905] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.238901 8457 ts_tablet_manager.cc:1905] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.241770 8457 ts_tablet_manager.cc:1918] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:46.241770 8455 ts_tablet_manager.cc:1918] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:46.241848 8455 log.cc:1196] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/188e16b478ca49a58d7577516c6d3c3e
I20260924 04:41:46.241848 8457 log.cc:1196] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/188e16b478ca49a58d7577516c6d3c3e
I20260924 04:41:46.241976 8456 ts_tablet_manager.cc:1918] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:46.242049 8456 log.cc:1196] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/188e16b478ca49a58d7577516c6d3c3e
I20260924 04:41:46.242215 8455 ts_tablet_manager.cc:1939] T 188e16b478ca49a58d7577516c6d3c3e P 1bbeb61d9e7e4ae18f931f809150a866: Deleting consensus metadata
I20260924 04:41:46.242311 8456 ts_tablet_manager.cc:1939] T 188e16b478ca49a58d7577516c6d3c3e P cca80a60690d496b9c442bb4f4affd68: Deleting consensus metadata
I20260924 04:41:46.242517 8457 ts_tablet_manager.cc:1939] T 188e16b478ca49a58d7577516c6d3c3e P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting consensus metadata
I20260924 04:41:46.243243 7906 catalog_manager.cc:4841] TS cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157): tablet 188e16b478ca49a58d7577516c6d3c3e (table test-table [id=f7c6b8e4e215483fa76aa4b2b755cfab]) successfully deleted
I20260924 04:41:46.243499 7908 catalog_manager.cc:4841] TS 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865): tablet 188e16b478ca49a58d7577516c6d3c3e (table test-table [id=f7c6b8e4e215483fa76aa4b2b755cfab]) successfully deleted
I20260924 04:41:46.243607 7908 catalog_manager.cc:4841] TS 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383): tablet 188e16b478ca49a58d7577516c6d3c3e (table test-table [id=f7c6b8e4e215483fa76aa4b2b755cfab]) successfully deleted
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:41:46.270061 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:46.262562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.0.98.62:37411)
0924 04:41:46.262727 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.262732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.262932 (+ 200us) server_negotiation.cc:407] Connection header received
0924 04:41:46.263084 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.263087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.263180 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.263266 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:46.264178 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.264745 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.266015 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.266230 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.266321 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.269914 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54338: BlockingRecv error: recv got EOF from 127.0.0.1:54338 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:41:46.281018 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.272020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.0.98.62:37411)
0924 04:41:46.272136 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.272142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.272279 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:41:46.272455 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.272458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.272511 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.272598 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:46.273208 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.273757 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.274620 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.274872 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.274974 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.277779 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:46.277782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:46.277785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:46.277813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:46.279368 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:46.279798 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:46.279805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:46.279808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:46.279889 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:46.280161 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:46.280165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:46.280167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:46.280359 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:46.280497 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.280735 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.280846 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:41:46.284067 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:46.284577 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:46.289884 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 564ee6f0a81f479d939c0f1a27ea32ed (DEFAULT_TABLE table=test-table [id=d187521711d648278e032cec7fb1ba77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:46.290014 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 564ee6f0a81f479d939c0f1a27ea32ed (DEFAULT_TABLE table=test-table [id=d187521711d648278e032cec7fb1ba77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:46.290251 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564ee6f0a81f479d939c0f1a27ea32ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:46.290287 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564ee6f0a81f479d939c0f1a27ea32ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:46.292922 8434 tablet_bootstrap.cc:492] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Bootstrap starting.
I20260924 04:41:46.293515 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 564ee6f0a81f479d939c0f1a27ea32ed (DEFAULT_TABLE table=test-table [id=d187521711d648278e032cec7fb1ba77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:46.293830 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564ee6f0a81f479d939c0f1a27ea32ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:46.293838 8434 tablet_bootstrap.cc:654] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:46.294395 8433 tablet_bootstrap.cc:492] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Bootstrap starting.
I20260924 04:41:46.295372 8433 tablet_bootstrap.cc:654] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:46.296165 8434 tablet_bootstrap.cc:492] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: No bootstrap required, opened a new log
I20260924 04:41:46.296267 8434 ts_tablet_manager.cc:1397] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:46.296485 8435 tablet_bootstrap.cc:492] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Bootstrap starting.
I20260924 04:41:46.296579 8433 tablet_bootstrap.cc:492] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: No bootstrap required, opened a new log
I20260924 04:41:46.296649 8433 ts_tablet_manager.cc:1397] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:46.296718 8434 raft_consensus.cc:348] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.296880 8434 raft_consensus.cc:374] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:46.296924 8434 raft_consensus.cc:729] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Initialized, Role: FOLLOWER
I20260924 04:41:46.297039 8433 raft_consensus.cc:348] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.297062 8434 consensus_queue.cc:260] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.297138 8433 raft_consensus.cc:374] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:46.297166 8433 raft_consensus.cc:729] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45465cc3524a68ad2f4b011ea1a5dd, State: Initialized, Role: FOLLOWER
I20260924 04:41:46.297271 8433 consensus_queue.cc:260] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.297377 8434 ts_tablet_manager.cc:1428] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:46.297425 8435 tablet_bootstrap.cc:654] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:46.297502 8433 ts_tablet_manager.cc:1428] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:46.299607 8435 tablet_bootstrap.cc:492] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: No bootstrap required, opened a new log
I20260924 04:41:46.299700 8435 ts_tablet_manager.cc:1397] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 04:41:46.300143 8435 raft_consensus.cc:348] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.300276 8435 raft_consensus.cc:374] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:46.300328 8435 raft_consensus.cc:729] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Initialized, Role: FOLLOWER
I20260924 04:41:46.300485 8435 consensus_queue.cc:260] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.300802 8435 ts_tablet_manager.cc:1428] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:41:46.355228 8480 raft_consensus.cc:482] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:46.355371 8480 raft_consensus.cc:504] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.355844 8480 leader_election.cc:288] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865), cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:46.356343 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564ee6f0a81f479d939c0f1a27ea32ed" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca80a60690d496b9c442bb4f4affd68" is_pre_election: true
I20260924 04:41:46.356331 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564ee6f0a81f479d939c0f1a27ea32ed" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" is_pre_election: true
I20260924 04:41:46.356477 8359 raft_consensus.cc:2454] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 0.
I20260924 04:41:46.356526 8219 raft_consensus.cc:2454] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 0.
I20260924 04:41:46.356781 8010 leader_election.cc:302] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:46.357017 8480 raft_consensus.cc:2781] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:46.357095 8480 raft_consensus.cc:482] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:46.357154 8480 raft_consensus.cc:3037] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.358526 8480 raft_consensus.cc:504] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.358932 8480 leader_election.cc:288] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865), cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:46.359304 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564ee6f0a81f479d939c0f1a27ea32ed" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd"
I20260924 04:41:46.359321 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564ee6f0a81f479d939c0f1a27ea32ed" candidate_uuid: "1bbeb61d9e7e4ae18f931f809150a866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca80a60690d496b9c442bb4f4affd68"
I20260924 04:41:46.359407 8219 raft_consensus.cc:3037] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.359457 8359 raft_consensus.cc:3037] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.360571 8359 raft_consensus.cc:2454] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 1.
I20260924 04:41:46.360570 8219 raft_consensus.cc:2454] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbeb61d9e7e4ae18f931f809150a866 in term 1.
I20260924 04:41:46.360852 8010 leader_election.cc:302] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, cca80a60690d496b9c442bb4f4affd68; no voters:
I20260924 04:41:46.361042 8480 raft_consensus.cc:2781] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:46.361129 8480 raft_consensus.cc:686] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 LEADER]: Becoming Leader. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Running, Role: LEADER
I20260924 04:41:46.361275 8480 consensus_queue.cc:237] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.362983 7920 catalog_manager.cc:5495] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1bbeb61d9e7e4ae18f931f809150a866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:46.383313 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.379908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.0.98.1:38383)
0924 04:41:46.380048 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.380053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.380069 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:41:46.380145 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.380148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.380195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.380269 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.380479 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.381267 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.382184 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.382418 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.382537 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.382789 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.382870 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.382955 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.383077 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.383163 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:41:46.385476 8219 raft_consensus.cc:1261] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Refusing update from remote peer 1bbeb61d9e7e4ae18f931f809150a866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.385490 8359 raft_consensus.cc:1261] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Refusing update from remote peer 1bbeb61d9e7e4ae18f931f809150a866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.385871 8480 consensus_queue.cc:1035] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.386008 8440 consensus_queue.cc:1035] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.397462 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:46.397603 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:46.399647 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Sending DeleteTablet for 3 replicas of tablet 564ee6f0a81f479d939c0f1a27ea32ed
I20260924 04:41:46.400322 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 564ee6f0a81f479d939c0f1a27ea32ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45372
I20260924 04:41:46.400410 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 564ee6f0a81f479d939c0f1a27ea32ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43810
I20260924 04:41:46.400516 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 564ee6f0a81f479d939c0f1a27ea32ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59010
I20260924 04:41:46.400599 8456 tablet_replica.cc:330] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: stopping tablet replica
I20260924 04:41:46.400694 8457 tablet_replica.cc:330] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: stopping tablet replica
I20260924 04:41:46.400749 8456 raft_consensus.cc:2229] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:46.400820 8457 raft_consensus.cc:2229] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:46.400897 8456 raft_consensus.cc:2258] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.400954 8457 raft_consensus.cc:2258] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.401129 8455 tablet_replica.cc:330] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: stopping tablet replica
I20260924 04:41:46.401266 8455 raft_consensus.cc:2229] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:46.401563 8455 raft_consensus.cc:2258] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.401829 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:41:46.402252 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:41:46.402273 8455 ts_tablet_manager.cc:1905] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.403452 8457 ts_tablet_manager.cc:1905] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.404724 8455 ts_tablet_manager.cc:1918] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:46.404801 8455 log.cc:1196] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/564ee6f0a81f479d939c0f1a27ea32ed
I20260924 04:41:46.405089 8455 ts_tablet_manager.cc:1939] T 564ee6f0a81f479d939c0f1a27ea32ed P 1bbeb61d9e7e4ae18f931f809150a866: Deleting consensus metadata
I20260924 04:41:46.405097 8456 ts_tablet_manager.cc:1905] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.405761 7908 catalog_manager.cc:4841] TS 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383): tablet 564ee6f0a81f479d939c0f1a27ea32ed (table test-table [id=d187521711d648278e032cec7fb1ba77]) successfully deleted
I20260924 04:41:46.406155 8457 ts_tablet_manager.cc:1918] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:46.406245 8457 log.cc:1196] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/564ee6f0a81f479d939c0f1a27ea32ed
I20260924 04:41:46.406507 8457 ts_tablet_manager.cc:1939] T 564ee6f0a81f479d939c0f1a27ea32ed P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting consensus metadata
I20260924 04:41:46.407225 7908 catalog_manager.cc:4841] TS 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865): tablet 564ee6f0a81f479d939c0f1a27ea32ed (table test-table [id=d187521711d648278e032cec7fb1ba77]) successfully deleted
I20260924 04:41:46.407516 8456 ts_tablet_manager.cc:1918] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:41:46.407608 8456 log.cc:1196] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/564ee6f0a81f479d939c0f1a27ea32ed
I20260924 04:41:46.407841 8456 ts_tablet_manager.cc:1939] T 564ee6f0a81f479d939c0f1a27ea32ed P cca80a60690d496b9c442bb4f4affd68: Deleting consensus metadata
I20260924 04:41:46.408514 7906 catalog_manager.cc:4841] TS cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157): tablet 564ee6f0a81f479d939c0f1a27ea32ed (table test-table [id=d187521711d648278e032cec7fb1ba77]) successfully deleted
I20260924 04:41:46.409682 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcf7df19cf84b78860118d3f68683a3 (DEFAULT_TABLE table=test-table [id=c189b1c8f6754bc29ab43bb809b470ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:46.409682 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcf7df19cf84b78860118d3f68683a3 (DEFAULT_TABLE table=test-table [id=c189b1c8f6754bc29ab43bb809b470ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:46.409922 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcf7df19cf84b78860118d3f68683a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:46.409997 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcf7df19cf84b78860118d3f68683a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:46.412187 8433 tablet_bootstrap.cc:492] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Bootstrap starting.
I20260924 04:41:46.412906 8433 tablet_bootstrap.cc:654] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:46.413527 8434 tablet_bootstrap.cc:492] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866: Bootstrap starting.
I20260924 04:41:46.414196 8433 tablet_bootstrap.cc:492] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: No bootstrap required, opened a new log
I20260924 04:41:46.414290 8433 ts_tablet_manager.cc:1397] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:46.414397 8434 tablet_bootstrap.cc:654] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:46.414706 8433 raft_consensus.cc:348] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.414826 8433 raft_consensus.cc:374] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:46.414862 8433 raft_consensus.cc:729] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45465cc3524a68ad2f4b011ea1a5dd, State: Initialized, Role: FOLLOWER
I20260924 04:41:46.414975 8433 consensus_queue.cc:260] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.415266 8433 ts_tablet_manager.cc:1428] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:46.415936 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcf7df19cf84b78860118d3f68683a3 (DEFAULT_TABLE table=test-table [id=c189b1c8f6754bc29ab43bb809b470ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:41:46.416061 8434 tablet_bootstrap.cc:492] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866: No bootstrap required, opened a new log
I20260924 04:41:46.416132 8434 ts_tablet_manager.cc:1397] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:46.416210 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcf7df19cf84b78860118d3f68683a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:46.416530 8434 raft_consensus.cc:348] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.416639 8434 raft_consensus.cc:374] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:46.416687 8434 raft_consensus.cc:729] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbeb61d9e7e4ae18f931f809150a866, State: Initialized, Role: FOLLOWER
I20260924 04:41:46.416813 8434 consensus_queue.cc:260] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.417047 8434 ts_tablet_manager.cc:1428] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:46.418697 8435 tablet_bootstrap.cc:492] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Bootstrap starting.
I20260924 04:41:46.419690 8435 tablet_bootstrap.cc:654] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:46.420819 8435 tablet_bootstrap.cc:492] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: No bootstrap required, opened a new log
I20260924 04:41:46.420904 8435 ts_tablet_manager.cc:1397] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:46.421281 8435 raft_consensus.cc:348] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.421399 8435 raft_consensus.cc:374] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:46.421432 8435 raft_consensus.cc:729] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca80a60690d496b9c442bb4f4affd68, State: Initialized, Role: FOLLOWER
I20260924 04:41:46.421551 8435 consensus_queue.cc:260] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.421792 8435 ts_tablet_manager.cc:1428] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:41:46.625941 8439 raft_consensus.cc:482] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:41:46.626075 8439 raft_consensus.cc:504] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.627314 8439 leader_election.cc:288] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383), cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:46.632536 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.627552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60019 (local address 127.0.98.1:38383)
0924 04:41:46.627684 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.627691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.628012 (+ 321us) server_negotiation.cc:407] Connection header received
0924 04:41:46.628137 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.628140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.628190 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.628278 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:46.628591 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.629391 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.631236 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.631865 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.631988 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.632082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.632349 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.632421 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:41:46.632557 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.627540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34157 (local address 127.0.98.2:60209)
0924 04:41:46.627870 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.627883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.628025 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.628302 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.628305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.628329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:46.628477 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.628482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.629543 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.629545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.631090 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.631101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.632042 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.632047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.632159 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.632213 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.632286 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.632386 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":173,"spinlock_wait_cycles":22144,"thread_start_us":74,"threads_started":1}
I20260924 04:41:46.632942 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.627459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.98.2:60019)
0924 04:41:46.627823 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.627849 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.628025 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.628302 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.628305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.628329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:41:46.628470 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.628476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.629540 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.629544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.631090 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.631101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.632104 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.632107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.632216 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.632232 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.632689 (+ 457us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.632761 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":144,"spinlock_wait_cycles":30592,"thread_start_us":87,"threads_started":1}
I20260924 04:41:46.633496 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.627664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60209 (local address 127.0.98.3:34157)
0924 04:41:46.627799 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.627802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.627984 (+ 182us) server_negotiation.cc:407] Connection header received
0924 04:41:46.628137 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.628140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.628190 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.628278 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:41:46.628591 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.629422 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.631235 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.631892 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.632004 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.633197 (+ 1193us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.633238 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.633342 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:41:46.633498 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcf7df19cf84b78860118d3f68683a3" candidate_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbeb61d9e7e4ae18f931f809150a866" is_pre_election: true
I20260924 04:41:46.633672 8079 raft_consensus.cc:2454] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45465cc3524a68ad2f4b011ea1a5dd in term 0.
I20260924 04:41:46.634076 8152 leader_election.cc:302] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, 9c45465cc3524a68ad2f4b011ea1a5dd; no voters:
I20260924 04:41:46.634136 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcf7df19cf84b78860118d3f68683a3" candidate_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca80a60690d496b9c442bb4f4affd68" is_pre_election: true
I20260924 04:41:46.634267 8359 raft_consensus.cc:2454] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c45465cc3524a68ad2f4b011ea1a5dd in term 0.
I20260924 04:41:46.634413 8439 raft_consensus.cc:2781] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:41:46.634478 8439 raft_consensus.cc:482] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:41:46.634528 8439 raft_consensus.cc:3037] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.636053 8439 raft_consensus.cc:504] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.636502 8439 leader_election.cc:288] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [CANDIDATE]: Term 1 election: Requested vote from peers 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383), cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157)
I20260924 04:41:46.637086 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcf7df19cf84b78860118d3f68683a3" candidate_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca80a60690d496b9c442bb4f4affd68"
I20260924 04:41:46.637253 8359 raft_consensus.cc:3037] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.637224 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcf7df19cf84b78860118d3f68683a3" candidate_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbeb61d9e7e4ae18f931f809150a866"
I20260924 04:41:46.637349 8079 raft_consensus.cc:3037] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:46.638422 8079 raft_consensus.cc:2454] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45465cc3524a68ad2f4b011ea1a5dd in term 1.
I20260924 04:41:46.638610 8359 raft_consensus.cc:2454] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c45465cc3524a68ad2f4b011ea1a5dd in term 1.
I20260924 04:41:46.638753 8152 leader_election.cc:302] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbeb61d9e7e4ae18f931f809150a866, 9c45465cc3524a68ad2f4b011ea1a5dd; no voters:
I20260924 04:41:46.638931 8439 raft_consensus.cc:2781] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:46.639186 8439 raft_consensus.cc:686] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 LEADER]: Becoming Leader. State: Replica: 9c45465cc3524a68ad2f4b011ea1a5dd, State: Running, Role: LEADER
I20260924 04:41:46.639567 8439 consensus_queue.cc:237] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } }
I20260924 04:41:46.642606 7919 catalog_manager.cc:5495] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c45465cc3524a68ad2f4b011ea1a5dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 } health_report { overall_health: UNKNOWN } } }
I20260924 04:41:46.658134 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.654445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.0.98.2:34865)
0924 04:41:46.654708 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.654714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.654726 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:46.654791 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.654799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.654849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.654997 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.655235 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.655748 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.656697 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.656968 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.657043 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.657296 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.657766 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.657863 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.657957 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.658005 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":130,"threads_started":1}
I20260924 04:41:46.668277 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.665235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48663 (local address 127.0.98.1:38383)
0924 04:41:46.665355 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.665358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.665506 (+ 148us) server_negotiation.cc:407] Connection header received
0924 04:41:46.665614 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.665616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.665659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.665724 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.665924 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.666432 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.667396 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.667610 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.667689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.667931 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.667997 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.668049 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.668123 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.668166 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:41:46.668337 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.665126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.98.2:48663)
0924 04:41:46.665431 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.665444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.665529 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.665740 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.665742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.665748 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:46.665840 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.665845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.666534 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.666536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.667266 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.667275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.667705 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.667707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.667776 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.667819 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.668064 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.668069 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.668116 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.668177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
I20260924 04:41:46.674130 8359 raft_consensus.cc:1261] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Refusing update from remote peer 9c45465cc3524a68ad2f4b011ea1a5dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.674142 8079 raft_consensus.cc:1261] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Refusing update from remote peer 9c45465cc3524a68ad2f4b011ea1a5dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:41:46.674629 8514 consensus_queue.cc:1035] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bbeb61d9e7e4ae18f931f809150a866" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.674829 8439 consensus_queue.cc:1035] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca80a60690d496b9c442bb4f4affd68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:41:46.692356 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.688592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.98.1:57365)
0924 04:41:46.688724 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.688741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:46.688833 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:46.689298 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:46.689300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:46.689307 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:41:46.689397 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.689403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.690298 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.690302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.690993 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:46.691002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.691486 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.691490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:46.691584 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.691610 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.692017 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.692024 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:41:46.692097 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:41:46.692190 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 04:41:46.692389 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.688691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57365 (local address 127.0.98.2:34865)
0924 04:41:46.688796 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:46.688803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:46.688821 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:41:46.689075 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:46.689083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:46.689157 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:46.689260 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:41:46.689500 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.690196 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.691108 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:46.691372 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:46.691490 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:46.691757 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:41:46.691918 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:41:46.691991 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:46.692196 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 04:41:46.692258 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:41:46.716161 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:46.716349 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:41:46.718670 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e81de2d4e264a85a330a797741ba1b7: Sending DeleteTablet for 3 replicas of tablet 6dcf7df19cf84b78860118d3f68683a3
I20260924 04:41:46.719380 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcf7df19cf84b78860118d3f68683a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43810
I20260924 04:41:46.719586 8456 tablet_replica.cc:330] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: stopping tablet replica
I20260924 04:41:46.719552 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcf7df19cf84b78860118d3f68683a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59010
I20260924 04:41:46.719696 8456 raft_consensus.cc:2229] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:46.719753 8455 tablet_replica.cc:330] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866: stopping tablet replica
I20260924 04:41:46.719825 8456 raft_consensus.cc:2258] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.719863 8455 raft_consensus.cc:2229] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:41:46.719976 8455 raft_consensus.cc:2258] T 6dcf7df19cf84b78860118d3f68683a3 P 1bbeb61d9e7e4ae18f931f809150a866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.720059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 7994
I20260924 04:41:46.721856 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcf7df19cf84b78860118d3f68683a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:41:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45372
I20260924 04:41:46.722069 8457 tablet_replica.cc:330] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: stopping tablet replica
I20260924 04:41:46.722227 8457 raft_consensus.cc:2229] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:41:46.722594 8457 raft_consensus.cc:2258] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:41:46.723789 8457 ts_tablet_manager.cc:1905] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:41:46.727211 8457 ts_tablet_manager.cc:1918] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:46.727322 8457 log.cc:1196] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/6dcf7df19cf84b78860118d3f68683a3
I20260924 04:41:46.727671 8457 ts_tablet_manager.cc:1939] T 6dcf7df19cf84b78860118d3f68683a3 P 9c45465cc3524a68ad2f4b011ea1a5dd: Deleting consensus metadata
I20260924 04:41:46.728559 7908 catalog_manager.cc:4841] TS 9c45465cc3524a68ad2f4b011ea1a5dd (127.0.98.2:34865): tablet 6dcf7df19cf84b78860118d3f68683a3 (table test-table [id=c189b1c8f6754bc29ab43bb809b470ff]) successfully deleted
I20260924 04:41:46.728994 8456 ts_tablet_manager.cc:1905] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:41:46.729810 7908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38383 (error 108)
I20260924 04:41:46.730085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 8134
I20260924 04:41:46.732100 8456 ts_tablet_manager.cc:1918] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:41:46.732189 8456 log.cc:1196] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/6dcf7df19cf84b78860118d3f68683a3
I20260924 04:41:46.732470 8456 ts_tablet_manager.cc:1939] T 6dcf7df19cf84b78860118d3f68683a3 P cca80a60690d496b9c442bb4f4affd68: Deleting consensus metadata
I20260924 04:41:46.733407 7906 catalog_manager.cc:4841] TS cca80a60690d496b9c442bb4f4affd68 (127.0.98.3:34157): tablet 6dcf7df19cf84b78860118d3f68683a3 (table test-table [id=c189b1c8f6754bc29ab43bb809b470ff]) successfully deleted
I20260924 04:41:46.733817 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:46.733419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38383 (local address 127.0.0.1:59038)
0924 04:41:46.733669 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:46.733730 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
W20260924 04:41:46.734032 7908 catalog_manager.cc:4584] TS 1bbeb61d9e7e4ae18f931f809150a866 (127.0.98.1:38383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dcf7df19cf84b78860118d3f68683a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38383: connect: Connection refused (error 111)
I20260924 04:41:46.737877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 8274
I20260924 04:41:46.744747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 7890
2026-09-24T04:41:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3290 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[8528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[8528](info): set up 2 sockets
Sep 24 04:41:46 dist-test-slave-npg2 krb5kdc[8528](info): commencing operation
krb5kdc: starting...
W20260924 04:41:47.796099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:47 dist-test-slave-npg2 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:47.926180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42075
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:48.040270 8541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:48.040630 8541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:48.040705 8541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:48.045142 8541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:48.045249 8541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:48.045286 8541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:48.045325 8541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:48.045363 8541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:48.051000 8541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:48.052256 8541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:48.053323 8541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:48.062286 8547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:48.062283 8549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:48.062321 8541 server_base.cc:1034] running on GCE node
W20260924 04:41:48.062688 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:48.063349 8541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:48.064622 8541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:48.065896 8541 hybrid_clock.cc:648] HybridClock initialized: now 1790224908065858 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:41:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1328 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:41:48 dist-test-slave-npg2 krb5kdc[8554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:41:48 dist-test-slave-npg2 krb5kdc[8554](info): set up 2 sockets
Sep 24 04:41:48 dist-test-slave-npg2 krb5kdc[8554](info): commencing operation
krb5kdc: starting...
W20260924 04:41:49.131910 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:41:49 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:41:49.259631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:49.377661 8567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:49.378077 8567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:49.378183 8567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:49.385546 8567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:49.385639 8567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:49.385682 8567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:49.385727 8567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:49.385767 8567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:49.391908 8567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:49.393040 8567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:49.394115 8567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:49.403226 8573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:49.403262 8572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:49.403254 8575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:49.403348 8567 server_base.cc:1034] running on GCE node
I20260924 04:41:49.404143 8567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:49.405224 8567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:49.406384 8567 hybrid_clock.cc:648] HybridClock initialized: now 1790224909406369 us; error 35 us; skew 500 ppm
Sep 24 04:41:49 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:49.415182 8567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:49.416350 8567 webserver.cc:460] Webserver started at http://127.0.98.62:44659/ using document root <none> and password file <none>
I20260924 04:41:49.416975 8567 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:49.417069 8567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:49.417311 8567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:49.419509 8567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "49c0702c8d4c43b39799135cf4df40ee"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "89e3e7cbbaa7c24f2c7ce2a37bfb4acd"
server_key_iv: "894eb7f8f3fb7944feeda9393ae827cf"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.420075 8567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "49c0702c8d4c43b39799135cf4df40ee"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "89e3e7cbbaa7c24f2c7ce2a37bfb4acd"
server_key_iv: "894eb7f8f3fb7944feeda9393ae827cf"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.423964 8567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:41:49.426549 8582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:49.428048 8567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:49.428198 8567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "49c0702c8d4c43b39799135cf4df40ee"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "89e3e7cbbaa7c24f2c7ce2a37bfb4acd"
server_key_iv: "894eb7f8f3fb7944feeda9393ae827cf"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.428319 8567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:49.467746 8567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:49.471339 8567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:49.471565 8567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:49.479225 8567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45427
I20260924 04:41:49.479223 8634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45427 every 8 connection(s)
I20260924 04:41:49.480255 8567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:49.483367 8635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:41:49.487308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 8567
I20260924 04:41:49.487479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:41:49.487807 392 external_mini_cluster.cc:1428] Setting key a3c9cde1908de8650656c88951d160e7
I20260924 04:41:49.490211 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Bootstrap starting.
I20260924 04:41:49.492682 8635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:41:49.493520 8635 log.cc:824] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:49.495772 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: No bootstrap required, opened a new log
Sep 24 04:41:49 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:49.499464 8635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:49.499727 8635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:41:49.499804 8635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c0702c8d4c43b39799135cf4df40ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:49.500376 8635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:49.500579 8635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:49.500653 8635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:49.500783 8635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:41:49.502028 8635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:49.502408 8635 leader_election.cc:302] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c0702c8d4c43b39799135cf4df40ee; no voters:
I20260924 04:41:49.502861 8635 leader_election.cc:288] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:41:49.503100 8640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:41:49.503381 8640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 LEADER]: Becoming Leader. State: Replica: 49c0702c8d4c43b39799135cf4df40ee, State: Running, Role: LEADER
I20260924 04:41:49.503719 8640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:49.504307 8635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:49.504379 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:49.489369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.0.98.62:45427)
0924 04:41:49.489867 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:49.489883 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:49.489921 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:41:49.490647 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:49.490666 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:49.491023 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:49.491416 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:49.492254 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:49.493383 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:49.494554 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:49.495033 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:49.495241 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:49.498097 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:49.498102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:49.498116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:49.498142 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:49.500974 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:49.501426 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:49.501433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:49.501439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:49.501598 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:49.501880 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:49.501887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:49.501892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:49.502475 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:49.502668 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:49.502963 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 04:41:49.503313 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
I20260924 04:41:49.504938 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c0702c8d4c43b39799135cf4df40ee. Latest consensus state: current_term: 1 leader_uuid: "49c0702c8d4c43b39799135cf4df40ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260924 04:41:49.505056 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:49.505328 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c0702c8d4c43b39799135cf4df40ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260924 04:41:49.505401 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:49.505458 8646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:49.508678 8646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:49.514113 8646 catalog_manager.cc:1347] Generated new cluster ID: 8d4455ea19e543959f94158a31ff3508
I20260924 04:41:49.514199 8646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:49.535445 8646 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:41:49.536494 8646 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:49.544296 8646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Generated new TSK 0
I20260924 04:41:49.545090 8646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:41:49.583946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:49.700831 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:49.701117 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:49.701161 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:49.705475 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:49.705541 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:49.705603 8661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:49.711031 8661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:49.712167 8661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:49.713169 8661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:49.723286 8669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:49.723309 8667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:49.723557 8661 server_base.cc:1034] running on GCE node
W20260924 04:41:49.723309 8666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:49.724329 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:49.725042 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:49.726280 8661 hybrid_clock.cc:648] HybridClock initialized: now 1790224909726225 us; error 48 us; skew 500 ppm
Sep 24 04:41:49 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:49.729328 8661 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:41:49.730289 8661 webserver.cc:460] Webserver started at http://127.0.98.1:42169/ using document root <none> and password file <none>
I20260924 04:41:49.730870 8661 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:49.731005 8661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:49.731298 8661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:49.733410 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf4c85c63aab4746bf001282da1f049f"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "7a199b4059b22f182ddceb6cff92cd65"
server_key_iv: "2b2d4c91c13bcaae6a1521a293e0255c"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.733990 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf4c85c63aab4746bf001282da1f049f"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "7a199b4059b22f182ddceb6cff92cd65"
server_key_iv: "2b2d4c91c13bcaae6a1521a293e0255c"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.737751 8661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:41:49.740260 8676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:49.741740 8661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:49.741901 8661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "cf4c85c63aab4746bf001282da1f049f"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "7a199b4059b22f182ddceb6cff92cd65"
server_key_iv: "2b2d4c91c13bcaae6a1521a293e0255c"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.742136 8661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:49.760046 8661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:49.763620 8661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:49.763827 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:49.764457 8661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:49.765375 8661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:49.765443 8661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:49.765492 8661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:49.765522 8661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:49.774499 8661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41127
I20260924 04:41:49.774518 8789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41127 every 8 connection(s)
I20260924 04:41:49.775488 8661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:49.780743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 8661
I20260924 04:41:49.780896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:41:49.781169 392 external_mini_cluster.cc:1428] Setting key 5033b16a7398053207f6c146d5b8e74f
Sep 24 04:41:49 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:41:49.790757 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:49.777530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41067 (local address 127.0.98.62:45427)
0924 04:41:49.777674 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:49.777677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:49.778477 (+ 800us) server_negotiation.cc:407] Connection header received
0924 04:41:49.780104 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:49.780108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:49.780161 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:49.780246 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:49.781600 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:49.782257 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:49.783612 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:49.784037 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:49.784158 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:49.787173 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:49.787179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:49.787182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:49.787211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:49.788661 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:49.789230 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:49.789234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:49.789236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:49.789316 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:49.789668 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:49.789673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:49.789675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:49.789860 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:49.789968 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:49.790463 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 04:41:49.790591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:41:49.791671 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:49.777812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:41067)
0924 04:41:49.778315 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:49.778359 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:49.779886 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:49.780430 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:49.780436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:49.780797 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:49.781445 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:49.781469 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:49.782441 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:49.782445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:49.783494 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:49.783503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:49.784205 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:49.784211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:49.784448 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:49.785144 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:49.785170 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:49.786978 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:49.788783 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:49.788789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:49.788805 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:49.789121 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:49.789419 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:49.789422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:49.789424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:49.789572 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:49.789969 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:49.789977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:49.790226 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:41:49.790418 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 04:41:49.790743 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260924 04:41:49.793099 8790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260924 04:41:49.793358 8790 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:41:49.794065 8790 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260924 04:41:49.795879 8599 ts_manager.cc:194] Registered new tserver with Master: cf4c85c63aab4746bf001282da1f049f (127.0.98.1:41127)
I20260924 04:41:49.797595 8599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:41:49.811200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:49.926746 8795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:49.927016 8795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:49.927058 8795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:49.931219 8795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:49.931277 8795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:49.931384 8795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:49.936919 8795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:49.937984 8795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:49.939043 8795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:49.949155 8800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:49.949144 8803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:49.949120 8801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:49.949368 8795 server_base.cc:1034] running on GCE node
I20260924 04:41:49.949843 8795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:49.950582 8795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:49.951776 8795 hybrid_clock.cc:648] HybridClock initialized: now 1790224909951764 us; error 36 us; skew 500 ppm
Sep 24 04:41:49 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:49.954653 8795 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:41:49.955698 8795 webserver.cc:460] Webserver started at http://127.0.98.2:34369/ using document root <none> and password file <none>
I20260924 04:41:49.956305 8795 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:49.956396 8795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:49.956631 8795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:49.958746 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "1cc66ae64fb2406fadc734fe66bd3244"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "b555335fbc31c0213455a65e4a0d62f8"
server_key_iv: "4bb89944a6ea78171f54bc426125ff59"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.959414 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1cc66ae64fb2406fadc734fe66bd3244"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "b555335fbc31c0213455a65e4a0d62f8"
server_key_iv: "4bb89944a6ea78171f54bc426125ff59"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.963188 8795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:41:49.965662 8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:49.967041 8795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:49.967198 8795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "1cc66ae64fb2406fadc734fe66bd3244"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "b555335fbc31c0213455a65e4a0d62f8"
server_key_iv: "4bb89944a6ea78171f54bc426125ff59"
server_key_version: "encryptionkey@0"
I20260924 04:41:49.967327 8795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:49.989226 8795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:49.992570 8795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:49.992733 8795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:49.993366 8795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:49.994148 8795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:49.994194 8795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:49.994232 8795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:49.994247 8795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.003961 8795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33669
I20260924 04:41:50.003984 8923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33669 every 8 connection(s)
I20260924 04:41:50.004989 8795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:50.008162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 8795
I20260924 04:41:50.008277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:41:50.008585 392 external_mini_cluster.cc:1428] Setting key 9f7f1975961bea0b1e7f8c74602748d2
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:41:50.021456 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.006933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59651 (local address 127.0.98.62:45427)
0924 04:41:50.007077 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.007080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.008239 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 04:41:50.009977 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.009981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.010033 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.010141 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.012030 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.013118 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.014501 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.014730 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.014849 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.017598 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:50.017601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:50.017603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:50.017629 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:50.019311 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:50.019862 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:50.019865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:50.019867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:50.019960 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:50.020316 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:50.020319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:50.020321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:50.020522 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:50.020627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:50.021155 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 04:41:50.021284 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:41:50.022363 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.007479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:59651)
0924 04:41:50.008031 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.008096 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.009697 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.010346 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.010356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.010891 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.011731 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.011753 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.013290 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.013293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.014251 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.014261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.014896 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.014900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.015106 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.015829 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.015847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.017389 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:50.019445 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:50.019452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:50.019471 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:50.019746 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:50.020056 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:50.020059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:50.020061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:50.020213 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:50.020625 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:50.020632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:50.020906 (+ 274us) client_negotiation.cc:769] Sending connection context
0924 04:41:50.021103 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 04:41:50.021484 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260924 04:41:50.023682 8924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260924 04:41:50.023973 8924 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:50.024511 8924 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260924 04:41:50.025593 8599 ts_manager.cc:194] Registered new tserver with Master: 1cc66ae64fb2406fadc734fe66bd3244 (127.0.98.2:33669)
I20260924 04:41:50.026175 8599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:41:50.039007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:50.153978 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:50.154290 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:50.154336 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:50.158787 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:50.158877 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:50.158975 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:50.164726 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:50.165880 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:50.166960 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:50.177438 8937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.177445 8935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.177445 8934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:50.177958 8929 server_base.cc:1034] running on GCE node
I20260924 04:41:50.178359 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:50.178987 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:50.180154 8929 hybrid_clock.cc:648] HybridClock initialized: now 1790224910180140 us; error 34 us; skew 500 ppm
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:50.183029 8929 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:41:50.183920 8929 webserver.cc:460] Webserver started at http://127.0.98.3:33793/ using document root <none> and password file <none>
I20260924 04:41:50.184448 8929 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:50.184509 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:50.184669 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:41:50.186682 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "38ecb68bcb9c43108ef583697292f37e"
format_stamp: "Formatted at 2026-09-24 04:41:50 on dist-test-slave-npg2"
server_key: "4c71e62e9c084200f8fa16495e311fd3"
server_key_iv: "21a9535de80b80e6517d6405df58ed1b"
server_key_version: "encryptionkey@0"
I20260924 04:41:50.187268 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38ecb68bcb9c43108ef583697292f37e"
format_stamp: "Formatted at 2026-09-24 04:41:50 on dist-test-slave-npg2"
server_key: "4c71e62e9c084200f8fa16495e311fd3"
server_key_iv: "21a9535de80b80e6517d6405df58ed1b"
server_key_version: "encryptionkey@0"
I20260924 04:41:50.190778 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:41:50.193395 8944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.194824 8929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:50.195029 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "38ecb68bcb9c43108ef583697292f37e"
format_stamp: "Formatted at 2026-09-24 04:41:50 on dist-test-slave-npg2"
server_key: "4c71e62e9c084200f8fa16495e311fd3"
server_key_iv: "21a9535de80b80e6517d6405df58ed1b"
server_key_version: "encryptionkey@0"
I20260924 04:41:50.195171 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:50.213032 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:50.216297 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:50.216456 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:50.216976 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:50.217968 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:50.218030 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.218101 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:50.218142 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.229779 8929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45151
I20260924 04:41:50.230039 9057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45151 every 8 connection(s)
I20260924 04:41:50.231002 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:50.235692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 8929
I20260924 04:41:50.235839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:41:50.236193 392 external_mini_cluster.cc:1428] Setting key 665bcc04b622682ad2d03c63741b35f9
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:41:50.253037 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.233545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40687 (local address 127.0.98.62:45427)
0924 04:41:50.233686 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.233691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.234694 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 04:41:50.236798 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.236801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.236868 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.236937 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.238679 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.239617 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.243398 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.243786 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.243879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.248007 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:50.248011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:50.248014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:50.248046 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:50.250191 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:50.250940 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:50.250946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:50.250950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:50.251057 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:50.251557 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:50.251564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:50.251567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:50.251849 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:50.251961 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:50.252650 (+ 689us) server_negotiation.cc:299] Negotiation successful
0924 04:41:50.252799 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:41:50.254137 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.233757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:40687)
0924 04:41:50.234507 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.234578 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.236580 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.237231 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.237238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.237762 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.238511 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.238532 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.239775 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.239778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.243188 (+ 3410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.243206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.244120 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.244125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.244359 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.245086 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.245103 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.247734 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:41:50.250347 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:50.250356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:50.250375 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:50.250793 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:50.251208 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:41:50.251211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:41:50.251213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:41:50.251434 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:41:50.251964 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:41:50.251971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:41:50.252320 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 04:41:50.252555 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 04:41:50.252944 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":260,"threads_started":1}
I20260924 04:41:50.255980 9058 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260924 04:41:50.256371 9058 heartbeater.cc:464] Registering TS with master...
I20260924 04:41:50.256919 9058 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260924 04:41:50.257948 8599 ts_manager.cc:194] Registered new tserver with Master: 38ecb68bcb9c43108ef583697292f37e (127.0.98.3:45151)
I20260924 04:41:50.258648 8599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40687
I20260924 04:41:50.266803 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:41:50.351284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 8661
I20260924 04:41:50.357014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 8795
I20260924 04:41:50.362277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 8929
I20260924 04:41:50.367591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 8567
I20260924 04:41:50.373908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--webserver_interface=127.0.98.62
--webserver_port=44659
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:41:50.486021 9067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:50.486325 9067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:50.486399 9067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:50.490518 9067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:41:50.490615 9067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:50.490650 9067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:50.490692 9067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:41:50.490731 9067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:41:50.496366 9067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:50.497449 9067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:50.498633 9067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:50.507620 9073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.507632 9072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:50.507730 9067 server_base.cc:1034] running on GCE node
W20260924 04:41:50.507654 9075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:50.508499 9067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:50.509670 9067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:50.510847 9067 hybrid_clock.cc:648] HybridClock initialized: now 1790224910510820 us; error 62 us; skew 500 ppm
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:50.514029 9067 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:41:50.514977 9067 webserver.cc:460] Webserver started at http://127.0.98.62:44659/ using document root <none> and password file <none>
I20260924 04:41:50.515599 9067 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:50.515699 9067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:50.519757 9067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:41:50.521938 9082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.523289 9067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:41:50.523434 9067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "49c0702c8d4c43b39799135cf4df40ee"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "89e3e7cbbaa7c24f2c7ce2a37bfb4acd"
server_key_iv: "894eb7f8f3fb7944feeda9393ae827cf"
server_key_version: "encryptionkey@0"
I20260924 04:41:50.523981 9067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:50.549147 9067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:50.552649 9067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:50.552990 9067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:50.560315 9067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45427
I20260924 04:41:50.560338 9134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45427 every 8 connection(s)
I20260924 04:41:50.561416 9067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:41:50.565879 9135 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:41:50.567734 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Bootstrap starting.
I20260924 04:41:50.570622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9067
I20260924 04:41:50.571193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41127
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45427
--webserver_port=42169
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:41:50.587088 9135 log.cc:824] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:41:50.594482 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:41:50.594952 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Bootstrap complete.
I20260924 04:41:50.599690 9135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:50.600148 9135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c0702c8d4c43b39799135cf4df40ee, State: Initialized, Role: FOLLOWER
I20260924 04:41:50.600658 9135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:50.600812 9135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:41:50.600901 9135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:41:50.601032 9135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:41:50.603048 9135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:50.603428 9135 leader_election.cc:302] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c0702c8d4c43b39799135cf4df40ee; no voters:
I20260924 04:41:50.603837 9135 leader_election.cc:288] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:41:50.603943 9139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:41:50.604146 9139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [term 2 LEADER]: Becoming Leader. State: Replica: 49c0702c8d4c43b39799135cf4df40ee, State: Running, Role: LEADER
I20260924 04:41:50.604462 9139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260924 04:41:50.605163 9135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:41:50.605815 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "49c0702c8d4c43b39799135cf4df40ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260924 04:41:50.605942 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:50.606664 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c0702c8d4c43b39799135cf4df40ee. Latest consensus state: current_term: 2 leader_uuid: "49c0702c8d4c43b39799135cf4df40ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c0702c8d4c43b39799135cf4df40ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260924 04:41:50.606812 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:41:50.606963 9146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:41:50.610292 9146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:41:50.611027 9146 catalog_manager.cc:1259] Loaded cluster ID: 8d4455ea19e543959f94158a31ff3508
I20260924 04:41:50.611093 9146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:41:50.613730 9146 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:41:50.614459 9146 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49c0702c8d4c43b39799135cf4df40ee: Loaded TSK: 0
I20260924 04:41:50.615787 9146 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:41:50.702728 9137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:50.703091 9137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:50.703182 9137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:50.707669 9137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:50.707754 9137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:50.707841 9137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:41:50.713783 9137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42169
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:50.714882 9137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:50.716037 9137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:50.726756 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.726763 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.727035 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:50.727602 9137 server_base.cc:1034] running on GCE node
I20260924 04:41:50.728097 9137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:50.728974 9137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:50.730181 9137 hybrid_clock.cc:648] HybridClock initialized: now 1790224910730161 us; error 29 us; skew 500 ppm
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:50.733866 9137 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:41:50.735109 9137 webserver.cc:460] Webserver started at http://127.0.98.1:42169/ using document root <none> and password file <none>
I20260924 04:41:50.735888 9137 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:50.735994 9137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:50.741068 9137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 04:41:50.743752 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.745357 9137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:41:50.745479 9137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "cf4c85c63aab4746bf001282da1f049f"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "7a199b4059b22f182ddceb6cff92cd65"
server_key_iv: "2b2d4c91c13bcaae6a1521a293e0255c"
server_key_version: "encryptionkey@0"
I20260924 04:41:50.746171 9137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:50.761574 9137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:50.765522 9137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:50.765715 9137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:50.766376 9137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:50.767395 9137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:50.767452 9137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.767501 9137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:50.767530 9137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.777740 9137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41127
I20260924 04:41:50.777763 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41127 every 8 connection(s)
I20260924 04:41:50.778977 9137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:41:50.784453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9137
I20260924 04:41:50.784981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33669
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45427
--webserver_port=34369
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:50.796433 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.781796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:56975)
0924 04:41:50.782379 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.782421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.783788 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.785595 (+ 1807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.785603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.786066 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.786670 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.786686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.788718 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.788722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.790088 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.790099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.791341 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.791345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.791576 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.792325 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.792346 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.795477 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":145,"threads_started":1}
W20260924 04:41:50.796868 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:50.781459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56975 (local address 127.0.98.62:45427)
0924 04:41:50.781869 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.781891 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.782548 (+ 657us) server_negotiation.cc:407] Connection header received
0924 04:41:50.784260 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.784277 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.784698 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.785067 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.786872 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.788567 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.790232 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.791166 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.791385 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.795808 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56975: BlockingRecv error: recv got EOF from 127.0.98.1:56975 (error 108)
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
W20260924 04:41:50.796943 9285 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45427 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45427: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:50.803265 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.797572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:33203)
0924 04:41:50.797694 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.797711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.797792 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.798044 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.798046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.798173 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.798267 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.798271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.799407 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.799410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.799957 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.799964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.800515 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.800517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.800583 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.801137 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.801150 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.803104 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:41:50.803483 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:50.797640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33203 (local address 127.0.98.62:45427)
0924 04:41:50.797744 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.797748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.797764 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:41:50.797883 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.797886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.797931 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.798022 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.798368 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.799275 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.800064 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.800408 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.800505 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.803333 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33203: BlockingRecv error: recv got EOF from 127.0.98.1:33203 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:50.809911 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.803844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:33297)
0924 04:41:50.804230 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.804244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.804346 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.804583 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.804586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.804789 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.804909 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.804915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.805654 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.805660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.806648 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.806659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.807250 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.807254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.807368 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.807862 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.807878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.809731 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
W20260924 04:41:50.809931 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:50.803900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33297 (local address 127.0.98.62:45427)
0924 04:41:50.804041 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.804045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.804329 (+ 284us) server_negotiation.cc:407] Connection header received
0924 04:41:50.804423 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.804426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.804474 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.804552 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.805021 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.805531 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.806755 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.807074 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.807236 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.809845 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33297: BlockingRecv error: recv got EOF from 127.0.98.1:33297 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:41:50.810135 9285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:41:50.904811 9289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:50.905118 9289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:50.905161 9289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:50.909305 9289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:50.909379 9289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:50.909438 9289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:41:50.914944 9289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34369
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:50.916117 9289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:50.917189 9289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:41:50.927393 9298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.927477 9295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:50.927505 9296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:50.927750 9289 server_base.cc:1034] running on GCE node
I20260924 04:41:50.928188 9289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:50.928879 9289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:50.930080 9289 hybrid_clock.cc:648] HybridClock initialized: now 1790224910930044 us; error 48 us; skew 500 ppm
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:50.933080 9289 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:41:50.934041 9289 webserver.cc:460] Webserver started at http://127.0.98.2:34369/ using document root <none> and password file <none>
I20260924 04:41:50.934623 9289 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:50.934719 9289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:50.938678 9289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:50.940979 9305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.942320 9289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:41:50.942495 9289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "1cc66ae64fb2406fadc734fe66bd3244"
format_stamp: "Formatted at 2026-09-24 04:41:49 on dist-test-slave-npg2"
server_key: "b555335fbc31c0213455a65e4a0d62f8"
server_key_iv: "4bb89944a6ea78171f54bc426125ff59"
server_key_version: "encryptionkey@0"
I20260924 04:41:50.943075 9289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:50.959205 9289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:50.962563 9289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:50.962790 9289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:50.963474 9289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:50.964445 9289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:50.964524 9289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.964589 9289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:50.964632 9289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:50.973992 9289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33669
I20260924 04:41:50.974020 9418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33669 every 8 connection(s)
I20260924 04:41:50.975064 9289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:41:50.982545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9289
I20260924 04:41:50.983067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45151
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45427
--webserver_port=33793
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:41:50.987226 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:50.976911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35505 (local address 127.0.98.62:45427)
0924 04:41:50.977064 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.977067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.978018 (+ 951us) server_negotiation.cc:407] Connection header received
0924 04:41:50.979369 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.979372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.979420 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.979495 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.980630 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.981186 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.982630 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.982971 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.983169 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.987063 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35505: BlockingRecv error: recv got EOF from 127.0.98.2:35505 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:41:50.988065 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.977356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:35505)
0924 04:41:50.977798 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.977839 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.979161 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.979626 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.979639 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.980004 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.980489 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.980502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.981330 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.981334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.982491 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.982506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.983287 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.983291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.983465 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.984119 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.984134 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.986910 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
W20260924 04:41:50.988582 9419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45427 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45427: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:50.994443 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.989112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:39541)
0924 04:41:50.989226 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.989242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.989351 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.989589 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.989592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.989762 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.989856 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.989862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.990588 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.990594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.991211 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.991220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.991656 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.991659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.991747 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.992320 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.992340 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:50.994318 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:41:50.994635 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:50.989237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39541 (local address 127.0.98.62:45427)
0924 04:41:50.989366 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.989371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.989397 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:41:50.989450 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.989452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.989487 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.989573 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.989969 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.990483 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.991319 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.991546 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.991654 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.994471 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39541: BlockingRecv error: recv got EOF from 127.0.98.2:39541 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:50 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:51.001050 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:50.995358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:41377)
0924 04:41:50.995600 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:50.995612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:50.995687 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:50.995943 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:50.995945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:50.996076 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:50.996145 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.996149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.997146 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.997149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.997741 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:50.997749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.998351 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.998354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:50.998424 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:50.998948 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:50.998960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:51.000952 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260924 04:41:51.001223 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:50.995493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41377 (local address 127.0.98.62:45427)
0924 04:41:50.995619 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:50.995623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:50.995657 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:41:50.995767 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:50.995770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:50.995811 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:50.995887 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:50.996250 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.997038 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.997850 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:50.998215 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:50.998315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.001111 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41377: BlockingRecv error: recv got EOF from 127.0.98.2:41377 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:41:51.001269 9419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:41:51.100095 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:41:51.100358 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:41:51.100402 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:41:51.104616 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:41:51.104681 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:41:51.104782 9422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:41:51.110782 9422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33793
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:41:51.111943 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:41:51.112982 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:41:51.123368 9422 server_base.cc:1034] running on GCE node
W20260924 04:41:51.123303 9430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:51.123535 9427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:41:51.123301 9428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:41:51.124024 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:41:51.124668 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:41:51.125892 9422 hybrid_clock.cc:648] HybridClock initialized: now 1790224911125865 us; error 40 us; skew 500 ppm
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224911, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:41:51.129016 9422 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:41:51.129978 9422 webserver.cc:460] Webserver started at http://127.0.98.3:33793/ using document root <none> and password file <none>
I20260924 04:41:51.130529 9422 fs_manager.cc:359] Metadata directory not provided
I20260924 04:41:51.130602 9422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:41:51.134642 9422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:41:51.136921 9437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:51.138249 9422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:41:51.138411 9422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "38ecb68bcb9c43108ef583697292f37e"
format_stamp: "Formatted at 2026-09-24 04:41:50 on dist-test-slave-npg2"
server_key: "4c71e62e9c084200f8fa16495e311fd3"
server_key_iv: "21a9535de80b80e6517d6405df58ed1b"
server_key_version: "encryptionkey@0"
I20260924 04:41:51.139051 9422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:41:51.159466 9422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:41:51.162617 9422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:41:51.162830 9422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:41:51.163527 9422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:41:51.164366 9422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:41:51.164439 9422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:51.164517 9422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:41:51.164561 9422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:41:51.173588 9422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45151
I20260924 04:41:51.173606 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45151 every 8 connection(s)
I20260924 04:41:51.174605 9422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:41:51.180703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9422
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:41:51.186398 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:51.176353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44097 (local address 127.0.98.62:45427)
0924 04:41:51.176510 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:51.176513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:51.177357 (+ 844us) server_negotiation.cc:407] Connection header received
0924 04:41:51.178703 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:51.178705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:51.178751 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:51.178842 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:51.180121 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.180627 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.182141 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.182437 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.182516 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.186216 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44097: BlockingRecv error: recv got EOF from 127.0.98.3:44097 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:41:51.186900 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:51.176758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:44097)
0924 04:41:51.177195 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:51.177237 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:51.178495 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:51.179055 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:51.179061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:51.179502 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:51.179988 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.180001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.180754 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.180760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.182029 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.182038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.182593 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.182598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.182824 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.183507 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:51.183523 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:51.186044 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
W20260924 04:41:51.187441 9551 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45427 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45427: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 04:41:51.191587 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:51.181764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.0.98.62:45427)
0924 04:41:51.182011 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:51.182014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:51.182025 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:41:51.182107 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:51.182112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:51.182160 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:51.182260 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:51.183284 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.183847 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.184838 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.185068 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.185173 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.186772 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:41:51.186778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:41:51.186792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:41:51.186818 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:41:51.188617 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:51.189710 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:51.189715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:51.189720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:51.189832 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:41:51.190184 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:41:51.190188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:41:51.190189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:41:51.190654 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:41:51.190811 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:41:51.191186 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 04:41:51.191446 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:51.193738 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:51.187933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:43035)
0924 04:41:51.188044 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:51.188059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:51.188132 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:51.188375 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:51.188378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:51.188536 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:51.188616 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.188621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.189308 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.189314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.190078 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.190088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.190598 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.190601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.190664 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.191194 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:51.191209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:51.193649 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:41:51.193917 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:51.187997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43035 (local address 127.0.98.62:45427)
0924 04:41:51.188108 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:51.188112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:51.188124 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:51.188238 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:51.188240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:51.188277 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:51.188361 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:51.188728 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.189205 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.190184 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.190485 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.190581 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.193798 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43035: BlockingRecv error: recv got EOF from 127.0.98.3:43035 (error 108)
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":13568}
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:51.199193 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:51.194516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:36665)
0924 04:41:51.194641 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:51.194656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:51.194731 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:51.194948 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:51.194951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:51.195099 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:51.195213 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.195223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.195919 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.195925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.196437 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.196445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.196932 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.196936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.197022 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.197453 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:51.197465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:51.199040 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 04:41:51.199256 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:51.194583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36665 (local address 127.0.98.62:45427)
0924 04:41:51.194721 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:51.194725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:51.194738 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:41:51.194808 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:51.194812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:51.194846 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:51.194936 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:51.195319 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.195816 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.196551 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.196800 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.196930 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.199156 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36665: BlockingRecv error: recv got EOF from 127.0.98.3:36665 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 04:41:51.199404 9551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:51 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:51.816560 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:51.810798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:32883)
0924 04:41:51.811091 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:51.811106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:51.811229 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:51.811551 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:51.811553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:51.811731 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:51.811843 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.811849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.812600 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.812602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.813313 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:51.813323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.813832 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.813835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:51.813951 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.814486 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:51.814498 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:51.816425 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
W20260924 04:41:51.816813 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:51.810948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32883 (local address 127.0.98.62:45427)
0924 04:41:51.811264 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:51.811268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:51.811282 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:41:51.811329 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:51.811331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:51.811399 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:51.811496 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:51.811956 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.812495 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.813442 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:51.813699 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:51.813831 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:51.816565 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32883: BlockingRecv error: recv got EOF from 127.0.98.1:32883 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
Sep 24 04:41:52 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:52 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:52.008226 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:52.001990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:52955)
0924 04:41:52.002370 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:52.002386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:52.002535 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:52.002829 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:52.002832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:52.003044 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:52.003179 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:52.003210 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.004043 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.004048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:52.004869 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:52.004880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.005313 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.005316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:52.005415 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:52.006101 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:52.006113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:52.008095 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":93,"threads_started":1}
W20260924 04:41:52.008436 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:52.002147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52955 (local address 127.0.98.62:45427)
0924 04:41:52.002337 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:52.002342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:52.002478 (+ 136us) server_negotiation.cc:407] Connection header received
0924 04:41:52.002671 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:52.002673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:52.002715 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:52.002808 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:52.003330 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.003910 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.004992 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.005209 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.005294 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:52.008259 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52955: BlockingRecv error: recv got EOF from 127.0.98.2:52955 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 04:41:52 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:52 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:52.206010 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:52.200076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:55643)
0924 04:41:52.200354 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:52.200367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:52.200519 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:52.200794 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:52.200797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:52.201010 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:52.201116 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:52.201122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.201880 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.201885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:52.202444 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:52.202451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.202921 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.202925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:52.203029 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:52.203676 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:52.203689 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:52.205847 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
W20260924 04:41:52.206162 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:52.200135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55643 (local address 127.0.98.62:45427)
0924 04:41:52.200304 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:52.200309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:52.200453 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:41:52.200618 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:52.200622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:52.200675 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:52.200771 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:52.201237 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.201768 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.202581 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.202801 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.202906 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:52.206027 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55643: BlockingRecv error: recv got EOF from 127.0.98.3:55643 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 04:41:52 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:52 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:52.822988 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:52.817460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:59441)
0924 04:41:52.817724 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:52.817741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:52.817858 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:52.818148 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:52.818151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:52.818327 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:52.818426 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:52.818432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.819301 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.819304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:52.819872 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:52.819880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.820317 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.820320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:52.820385 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:52.821029 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:52.821041 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:52.822853 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
W20260924 04:41:52.823321 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:52.817484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59441 (local address 127.0.98.62:45427)
0924 04:41:52.817716 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:52.817723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:52.817828 (+ 105us) server_negotiation.cc:407] Connection header received
0924 04:41:52.817955 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:52.817958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:52.818012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:52.818105 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:52.818528 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.819208 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.819980 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:52.820206 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:52.820282 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:52.823191 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59441: BlockingRecv error: recv got EOF from 127.0.98.1:59441 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
Sep 24 04:41:53 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:53 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:53.015257 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:53.009041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:34321)
0924 04:41:53.009476 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:53.009498 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:53.009642 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:53.009918 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:53.009921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:53.010171 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:53.010268 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:53.010274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.011067 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.011072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:53.011716 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:53.011724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.012196 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.012199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:53.012279 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:53.012838 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:53.012850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:53.014973 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":113,"threads_started":1}
W20260924 04:41:53.015268 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:53.009189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34321 (local address 127.0.98.62:45427)
0924 04:41:53.009447 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:53.009452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:53.009578 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:41:53.009742 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:53.009748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:53.009800 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:53.009890 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:53.010392 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.010948 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.011844 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.012074 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.012190 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:53.015164 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34321: BlockingRecv error: recv got EOF from 127.0.98.2:34321 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 24 04:41:53 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:53 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:53.213172 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:53.207072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:52561)
0924 04:41:53.207381 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:53.207396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:53.207484 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:53.207855 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:53.207857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:53.208057 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:53.208178 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:53.208183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.209064 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.209067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:53.209625 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:53.209632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.210138 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.210141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:53.210207 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:53.210748 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:53.210759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:53.212991 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":135,"threads_started":1}
W20260924 04:41:53.213215 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:53.207312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52561 (local address 127.0.98.62:45427)
0924 04:41:53.207473 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:53.207477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:53.207495 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:41:53.207672 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:53.207674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:53.207723 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:53.207813 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:53.208301 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.208882 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.209760 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.210020 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.210116 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:53.213118 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52561: BlockingRecv error: recv got EOF from 127.0.98.3:52561 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 04:41:53 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:53 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:53.830904 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:53.824008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:60051)
0924 04:41:53.824390 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:53.824414 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:53.824561 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:53.825052 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:53.825055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:53.825315 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:53.825454 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:53.825461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.826440 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.826443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:53.827190 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:53.827200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.827726 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.827729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:53.827801 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:53.828477 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:53.828490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:53.830770 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
W20260924 04:41:53.831210 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:53.824136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60051 (local address 127.0.98.62:45427)
0924 04:41:53.824520 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:53.824532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:53.824550 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:41:53.824728 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:53.824732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:53.824814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:53.824931 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:53.825629 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.826313 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.827303 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:53.827595 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:53.827692 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:53.830919 (+ 3227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60051: BlockingRecv error: recv got EOF from 127.0.98.1:60051 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
Sep 24 04:41:54 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:54 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:54.021910 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:54.016187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:47135)
0924 04:41:54.016534 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:54.016546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:54.016683 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:54.016959 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:54.016962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:54.017223 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:54.017317 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:54.017323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.018083 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.018087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:54.018663 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:54.018671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.019172 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.019176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:54.019263 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:54.019827 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:54.019848 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:54.021725 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":99,"threads_started":1}
W20260924 04:41:54.021976 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:54.016267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47135 (local address 127.0.98.62:45427)
0924 04:41:54.016419 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:54.016425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:54.016630 (+ 205us) server_negotiation.cc:407] Connection header received
0924 04:41:54.016791 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:54.016796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:54.016848 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:54.016937 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:54.017434 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.017964 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.018785 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.019022 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.019134 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:54.021867 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47135: BlockingRecv error: recv got EOF from 127.0.98.2:47135 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 04:41:54 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:54 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:54.220037 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:54.214024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:51771)
0924 04:41:54.214349 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:54.214366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:54.214540 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:54.214838 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:54.214841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:54.215052 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:54.215173 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:54.215179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.215963 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.215970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:54.216565 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:54.216573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.217039 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.217042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:54.217126 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:54.217697 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:54.217708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:54.219855 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
W20260924 04:41:54.220227 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:54.214203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51771 (local address 127.0.98.62:45427)
0924 04:41:54.214350 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:54.214355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:54.214458 (+ 103us) server_negotiation.cc:407] Connection header received
0924 04:41:54.214651 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:54.214655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:54.214711 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:54.214804 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:54.215298 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.215842 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.216700 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.216919 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.217028 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:54.220067 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51771: BlockingRecv error: recv got EOF from 127.0.98.3:51771 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 04:41:54 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:54 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:54.842985 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:54.831953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:58027)
0924 04:41:54.832380 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:54.832402 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:54.832499 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:54.832845 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:54.832851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:54.833149 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:54.833261 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:54.833267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.833991 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.833994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:54.834604 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:54.834612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.835068 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.835071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:54.835180 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:54.840726 (+ 5546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:54.840740 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:54.842833 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":105,"threads_started":1}
W20260924 04:41:54.843411 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:54.831990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58027 (local address 127.0.98.62:45427)
0924 04:41:54.832345 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:54.832350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:54.832491 (+ 141us) server_negotiation.cc:407] Connection header received
0924 04:41:54.832656 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:54.832658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:54.832708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:54.832791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:54.833368 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.833896 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.834715 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:54.834940 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:54.835063 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:54.843246 (+ 8183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58027: BlockingRecv error: recv got EOF from 127.0.98.1:58027 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
Sep 24 04:41:55 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:55 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:55.029305 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:55.023114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:49841)
0924 04:41:55.023431 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:55.023444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:55.023574 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:55.024066 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:55.024070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:55.024330 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:55.024483 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:55.024493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.025256 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.025259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:55.025868 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:55.025877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.026385 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.026388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:55.026459 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:55.027028 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:55.027040 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:55.029186 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
W20260924 04:41:55.029597 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:55.023203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49841 (local address 127.0.98.62:45427)
0924 04:41:55.023346 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:55.023349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:55.023499 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:41:55.023715 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:55.023718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:55.023798 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:55.024004 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:55.024613 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.025142 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.026015 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.026239 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.026329 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:55.029335 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49841: BlockingRecv error: recv got EOF from 127.0.98.2:49841 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 04:41:55 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:55 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:55.227442 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:55.220859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:54045)
0924 04:41:55.221226 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:55.221242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:55.221410 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:55.221709 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:55.221712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:55.221931 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:55.222024 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:55.222030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.223215 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.223218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:55.223784 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:55.223792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.224356 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.224360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:55.224454 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:55.225127 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:55.225145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:55.227241 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
W20260924 04:41:55.227471 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:55.221003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54045 (local address 127.0.98.62:45427)
0924 04:41:55.221191 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:55.221196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:55.221341 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:41:55.221548 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:55.221551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:55.221596 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:55.221686 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:55.222153 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.223016 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.223923 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.224186 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.224300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:55.227368 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54045: BlockingRecv error: recv got EOF from 127.0.98.3:54045 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 04:41:55 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:55 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:55.849809 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:55.844071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:43403)
0924 04:41:55.844313 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:55.844336 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:55.844452 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:55.844726 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:55.844729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:55.844907 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:55.845003 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:55.845008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.845823 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.845826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:55.846469 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:55.846480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.846931 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.846934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:55.846996 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:55.847604 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:55.847616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:55.849625 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
W20260924 04:41:55.849936 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:55.844016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43403 (local address 127.0.98.62:45427)
0924 04:41:55.844312 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:55.844316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:55.844404 (+ 88us) server_negotiation.cc:407] Connection header received
0924 04:41:55.844532 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:55.844535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:55.844576 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:55.844660 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:55.845126 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.845719 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.846609 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:55.846827 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:55.846907 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:55.849771 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43403: BlockingRecv error: recv got EOF from 127.0.98.1:43403 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":128,"threads_started":1}
Sep 24 04:41:56 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:56 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:56.036147 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:56.030156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:60469)
0924 04:41:56.030496 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:56.030514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:56.030631 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:56.030928 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:56.030931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:56.031166 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:56.031257 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:56.031263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.032084 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.032088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:56.032685 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:56.032693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.033233 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.033236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:56.033330 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:56.033898 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:56.033909 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:56.036008 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
W20260924 04:41:56.036288 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:56.030354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60469 (local address 127.0.98.62:45427)
0924 04:41:56.030541 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:56.030547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:56.030569 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:41:56.030755 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:56.030763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:56.030806 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:56.030895 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:56.031375 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.031940 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.032876 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.033112 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.033230 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:56.036165 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60469: BlockingRecv error: recv got EOF from 127.0.98.2:60469 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 04:41:56 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:56 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:56.234436 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:56.228461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:32779)
0924 04:41:56.228775 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:56.228788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:56.228890 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:56.229157 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:56.229159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:56.229353 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:56.229449 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:56.229454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.230335 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.230338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:56.230929 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:56.230937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.231400 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.231405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:56.231494 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:56.232114 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:56.232131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:56.234321 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":124,"threads_started":1}
W20260924 04:41:56.234720 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:56.228514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32779 (local address 127.0.98.62:45427)
0924 04:41:56.228670 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:56.228675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:56.228892 (+ 217us) server_negotiation.cc:407] Connection header received
0924 04:41:56.228968 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:56.228970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:56.229017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:56.229092 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:56.229589 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.230213 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.231045 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.231286 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.231368 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:56.234508 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32779: BlockingRecv error: recv got EOF from 127.0.98.3:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 04:41:56 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:56 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:56.856352 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:56.850685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:45415)
0924 04:41:56.850977 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:56.851000 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:56.851104 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:56.851408 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:56.851411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:56.851603 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:56.851710 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:56.851716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.852505 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.852509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:56.853141 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:56.853151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.853592 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.853595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:56.853691 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:56.854262 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:56.854280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:56.856230 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":115,"threads_started":1}
W20260924 04:41:56.856541 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:56.850757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45415 (local address 127.0.98.62:45427)
0924 04:41:56.851011 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:56.851015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:56.851066 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:41:56.851245 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:56.851247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:56.851290 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:56.851376 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:56.851792 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.852395 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.853257 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:56.853482 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:56.853571 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:56.856374 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45415: BlockingRecv error: recv got EOF from 127.0.98.1:45415 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
Sep 24 04:41:57 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:57 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:57.042871 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:57.037096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:48379)
0924 04:41:57.037414 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:57.037428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:57.037588 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:57.037858 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:57.037861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:57.038070 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:57.038159 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:57.038164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.038924 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.038928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:57.039617 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:57.039626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.040126 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.040130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:57.040210 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:57.040766 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:57.040778 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:57.042699 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
W20260924 04:41:57.042930 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:57.037233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48379 (local address 127.0.98.62:45427)
0924 04:41:57.037390 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:57.037396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:57.037513 (+ 117us) server_negotiation.cc:407] Connection header received
0924 04:41:57.037693 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:57.037698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:57.037748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:57.037841 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:57.038275 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.038797 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.039751 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.039973 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.040110 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:57.042822 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48379: BlockingRecv error: recv got EOF from 127.0.98.2:48379 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 04:41:57 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:57 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:57.240901 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:57.235261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:36503)
0924 04:41:57.235523 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:57.235537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:57.235644 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:57.235900 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:57.235903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:57.236086 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:57.236180 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:57.236185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.236902 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.236904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:57.237483 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:57.237492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.237933 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.237936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:57.237997 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:57.238514 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:57.238527 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:57.240727 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
W20260924 04:41:57.241012 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:57.235324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36503 (local address 127.0.98.62:45427)
0924 04:41:57.235475 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:57.235480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:57.235664 (+ 184us) server_negotiation.cc:407] Connection header received
0924 04:41:57.235713 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:57.235715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:57.235761 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:57.235838 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:57.236284 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.236805 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.237609 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.237830 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.237916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:57.240885 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36503: BlockingRecv error: recv got EOF from 127.0.98.3:36503 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 04:41:57 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:57 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:57.863524 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:57.857261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:39647)
0924 04:41:57.857635 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:57.857658 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:57.857750 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:57.858166 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:57.858169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:57.858364 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:57.858477 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:57.858483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.859390 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.859392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:57.859986 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:57.859995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.860461 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.860464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:57.860546 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:57.861168 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:57.861182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:57.863395 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":152,"threads_started":1}
W20260924 04:41:57.863716 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:57.857422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39647 (local address 127.0.98.62:45427)
0924 04:41:57.857805 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:57.857812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:57.857833 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:41:57.857874 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:57.857876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:57.857919 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:57.858090 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:57.858634 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.859253 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.860102 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:57.860324 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:57.860414 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:57.863566 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39647: BlockingRecv error: recv got EOF from 127.0.98.1:39647 (error 108)
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":237,"threads_started":1}
Sep 24 04:41:58 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:58 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:58.050041 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:58.043738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:42305)
0924 04:41:58.044173 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:58.044189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:58.044347 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:58.044646 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:58.044649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:58.044881 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:58.044979 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:58.044985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.045811 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.045816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:58.046665 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:58.046674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.047209 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.047212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:58.047302 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:58.047840 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:58.047852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:58.049915 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":87,"threads_started":1}
W20260924 04:41:58.050249 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:58.043865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42305 (local address 127.0.98.62:45427)
0924 04:41:58.044105 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:58.044111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:58.044280 (+ 169us) server_negotiation.cc:407] Connection header received
0924 04:41:58.044462 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:58.044466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:58.044520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:58.044615 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:58.045124 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.045690 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.046791 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.047033 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.047193 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:58.050113 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42305: BlockingRecv error: recv got EOF from 127.0.98.2:42305 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 24 04:41:58 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:58 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:58.248415 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:58.241791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:55617)
0924 04:41:58.242094 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:58.242112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:58.242228 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:58.242633 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:58.242636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:58.242813 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:58.242907 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:58.242913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.243939 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.243942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:58.244558 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:58.244568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.245345 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.245348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:58.245418 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:58.245975 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:58.245987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:58.248296 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
W20260924 04:41:58.248647 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:58.241936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55617 (local address 127.0.98.62:45427)
0924 04:41:58.242075 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:58.242078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:58.242219 (+ 141us) server_negotiation.cc:407] Connection header received
0924 04:41:58.242345 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:58.242350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:58.242413 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:58.242551 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:58.243091 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.243827 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.244688 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.244915 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.245101 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:58.248449 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55617: BlockingRecv error: recv got EOF from 127.0.98.3:55617 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 04:41:58 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:58 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:58.870404 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:58.864415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:43937)
0924 04:41:58.864704 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:58.864717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:58.864803 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:58.865112 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:58.865114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:58.865291 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:58.865384 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:58.865389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.866190 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.866193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:58.866784 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:58.866792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.867304 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.867307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:58.867375 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:58.868126 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:58.868139 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:58.870265 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
W20260924 04:41:58.870570 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:58.864564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43937 (local address 127.0.98.62:45427)
0924 04:41:58.864895 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:58.864899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:58.864911 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:41:58.864946 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:58.864948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:58.864987 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:58.865088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:58.865495 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.866080 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.866894 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:58.867163 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:58.867290 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:58.870413 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43937: BlockingRecv error: recv got EOF from 127.0.98.1:43937 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":186,"threads_started":1}
Sep 24 04:41:59 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:59 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:59.057135 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:59.050924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:42585)
0924 04:41:59.051284 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:59.051331 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:59.051454 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:59.051709 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:59.051712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:59.051898 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:59.051994 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:59.051999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.053082 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.053087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:59.053832 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:59.053845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.054339 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.054343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:59.054425 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:59.054913 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:59.054924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:59.057003 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":128,"threads_started":1}
W20260924 04:41:59.057487 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:59.051174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42585 (local address 127.0.98.62:45427)
0924 04:41:59.051291 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:59.051295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:59.051428 (+ 133us) server_negotiation.cc:407] Connection header received
0924 04:41:59.051561 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:59.051563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:59.051609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:59.051678 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:59.052176 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.052986 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.053964 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.054220 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.054338 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:59.057384 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42585: BlockingRecv error: recv got EOF from 127.0.98.2:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 04:41:59 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:59 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:59.255822 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:59.249299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:34713)
0924 04:41:59.249668 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:59.249684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:59.249879 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:59.250296 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:59.250299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:59.250525 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:59.250654 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:59.250661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.251643 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.251648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:59.252278 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:59.252287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.252857 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.252861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:59.252955 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:59.253488 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:59.253500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:59.255579 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":91,"threads_started":1}
W20260924 04:41:59.255843 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:59.249422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34713 (local address 127.0.98.62:45427)
0924 04:41:59.249617 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:59.249622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:59.249782 (+ 160us) server_negotiation.cc:407] Connection header received
0924 04:41:59.249999 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:59.250002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:59.250163 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:59.250273 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:59.250784 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.251513 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.252421 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.252728 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.252852 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:59.255735 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34713: BlockingRecv error: recv got EOF from 127.0.98.3:34713 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 04:41:59 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:41:59 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:41:59.884050 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:41:59.871434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:46967)
0924 04:41:59.871764 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:41:59.871778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:41:59.871877 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:41:59.872114 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:41:59.872116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:41:59.872295 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:41:59.872389 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:59.872394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.873252 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.873255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:59.874258 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:41:59.874269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.874835 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.874838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:41:59.874906 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:59.881739 (+ 6833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:41:59.881756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:41:59.883899 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":77,"threads_started":1}
W20260924 04:41:59.884311 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:41:59.871582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46967 (local address 127.0.98.62:45427)
0924 04:41:59.871867 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 04:41:59.871870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:41:59.871886 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:41:59.871947 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:41:59.871949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:41:59.871989 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:41:59.872077 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:41:59.872502 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.873091 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.874431 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:41:59.874702 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:41:59.874783 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:41:59.884130 (+ 9347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46967: BlockingRecv error: recv got EOF from 127.0.98.1:46967 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
Sep 24 04:42:00 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:00 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:00.072670 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:00.057888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:49743)
0924 04:42:00.058285 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:00.058304 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:00.058399 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:00.058706 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:00.058709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:00.058898 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:00.059226 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:00.059235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.060069 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.060072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:00.060629 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:00.060637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.061091 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.061094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:00.061157 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:00.070360 (+ 9203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:00.070381 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:00.072522 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":200,"threads_started":1}
W20260924 04:42:00.072876 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:00.058013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49743 (local address 127.0.98.62:45427)
0924 04:42:00.058191 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:00.058195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:00.058375 (+ 180us) server_negotiation.cc:407] Connection header received
0924 04:42:00.058479 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:00.058481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:00.058525 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:00.058596 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:00.059367 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.059896 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.060759 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.060977 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.061058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:00.072730 (+ 11672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49743: BlockingRecv error: recv got EOF from 127.0.98.2:49743 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:42:00 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:00 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:00.263284 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:00.256635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:50765)
0924 04:42:00.256978 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:00.256992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:00.257132 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:00.257422 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:00.257425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:00.257663 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:00.257761 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:00.257767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.258836 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.258841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:00.259620 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:00.259629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.260375 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.260379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:00.260465 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:00.261042 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:00.261056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:00.263096 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
W20260924 04:42:00.263446 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:00.256778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50765 (local address 127.0.98.62:45427)
0924 04:42:00.256937 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:00.256942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:00.257079 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:42:00.257250 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:00.257254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:00.257312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:00.257399 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:00.257884 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.258740 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.259894 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.260243 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.260369 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:00.263339 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50765: BlockingRecv error: recv got EOF from 127.0.98.3:50765 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 04:42:00 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:00 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:00.890338 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:00.885110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:39927)
0924 04:42:00.885421 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:00.885435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:00.885535 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:00.885791 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:00.885793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:00.885973 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:00.886067 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:00.886072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.886822 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.886824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:00.887448 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:00.887457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.887870 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.887872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:00.887931 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:00.888427 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:00.888438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:00.890220 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":117,"threads_started":1}
W20260924 04:42:00.890496 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:00.885191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39927 (local address 127.0.98.62:45427)
0924 04:42:00.885448 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:00.885452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:00.885495 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:42:00.885619 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:00.885621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:00.885663 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:00.885749 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:00.886201 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.886728 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.887545 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:00.887771 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:00.887851 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:00.890361 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39927: BlockingRecv error: recv got EOF from 127.0.98.1:39927 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
Sep 24 04:42:01 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:01 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:01.079854 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:01.073749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:55691)
0924 04:42:01.074034 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:01.074051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:01.074174 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:01.074408 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:01.074410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:01.074619 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:01.074725 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:01.074731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.075635 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.075637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:01.076215 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:01.076224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.076649 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.076651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:01.076714 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:01.077337 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:01.077349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:01.079698 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":111,"threads_started":1}
W20260924 04:42:01.080008 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:01.073819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55691 (local address 127.0.98.62:45427)
0924 04:42:01.073990 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:01.073994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:01.074134 (+ 140us) server_negotiation.cc:407] Connection header received
0924 04:42:01.074249 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:01.074251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:01.074293 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:01.074365 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:01.074835 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.075517 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.076327 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.076538 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.076628 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:01.079865 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55691: BlockingRecv error: recv got EOF from 127.0.98.2:55691 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 04:42:01 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:01 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:01.270031 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:01.264216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:55753)
0924 04:42:01.264504 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:01.264519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:01.264652 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:01.264951 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:01.264954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:01.265232 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:01.265328 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:01.265334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.266140 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.266142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:01.266726 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:01.266733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.267244 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.267248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:01.267317 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:01.267871 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:01.267882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:01.269846 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
W20260924 04:42:01.270094 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:01.264396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55753 (local address 127.0.98.62:45427)
0924 04:42:01.264539 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:01.264542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:01.264605 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:42:01.264744 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:01.264746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:01.264790 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:01.264873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:01.265461 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.266029 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.266841 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.267055 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.267207 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:01.269985 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55753: BlockingRecv error: recv got EOF from 127.0.98.3:55753 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:42:01 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:01 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:01.897055 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:01.891330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:58787)
0924 04:42:01.891673 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:01.891689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:01.891787 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:01.892153 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:01.892160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:01.892339 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:01.892435 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:01.892440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.893286 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.893288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:01.893887 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:01.893895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.894350 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.894352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:01.894412 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:01.894886 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:01.894897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:01.896939 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
W20260924 04:42:01.897243 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:01.891370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58787 (local address 127.0.98.62:45427)
0924 04:42:01.891721 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:01.891726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:01.891742 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:42:01.891936 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:01.891939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:01.892000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:01.892096 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:01.892581 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.893173 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.894003 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:01.894247 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:01.894335 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:01.897094 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58787: BlockingRecv error: recv got EOF from 127.0.98.1:58787 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":63,"threads_started":1}
Sep 24 04:42:02 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:02 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:02.088457 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:02.081009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:38331)
0924 04:42:02.081288 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:02.081303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:02.081422 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:02.081796 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:02.081799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:02.082071 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:02.082175 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:02.082182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.083475 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.083478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:02.084121 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:02.084131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.084861 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.084865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:02.085167 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:02.085814 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:02.085836 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:02.088233 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
W20260924 04:42:02.088479 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:02.081119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38331 (local address 127.0.98.62:45427)
0924 04:42:02.081296 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:02.081303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:02.081402 (+ 99us) server_negotiation.cc:407] Connection header received
0924 04:42:02.081531 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:02.081537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:02.081605 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:02.081725 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:02.082313 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.083330 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.084246 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.084707 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.084831 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:02.088388 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38331: BlockingRecv error: recv got EOF from 127.0.98.2:38331 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 04:42:02 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:02 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:02.277509 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:02.271087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:59429)
0924 04:42:02.271467 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:02.271481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:02.271598 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:02.271874 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:02.271876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:02.272054 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:02.272238 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:02.272246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.272988 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.272991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:02.273627 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:02.273635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.274057 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.274060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:02.274166 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:02.274829 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:02.274877 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:02.277334 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":97,"threads_started":1}
W20260924 04:42:02.277565 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:02.271347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59429 (local address 127.0.98.62:45427)
0924 04:42:02.271504 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:02.271508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:02.271541 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:42:02.271711 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:02.271714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:02.271761 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:02.271836 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:02.272347 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.272883 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.273740 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.273955 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.274036 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:02.277458 (+ 3422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59429: BlockingRecv error: recv got EOF from 127.0.98.3:59429 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 04:42:02 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:02 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:02.903769 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:02.898011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:44085)
0924 04:42:02.898288 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:02.898301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:02.898388 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:02.898601 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:02.898603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:02.898776 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:02.898867 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:02.898872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.899769 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.899772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:02.900388 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:02.900397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.900877 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.900880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:02.900946 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:02.901569 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:02.901581 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:02.903588 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
W20260924 04:42:02.903867 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:02.898146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44085 (local address 127.0.98.62:45427)
0924 04:42:02.898387 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:02.898391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:02.898404 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:42:02.898458 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:02.898461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:02.898508 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:02.898592 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:02.899005 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.899634 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.900510 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:02.900768 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:02.900854 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:02.903734 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44085: BlockingRecv error: recv got EOF from 127.0.98.1:44085 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
Sep 24 04:42:03 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:03 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:03.095353 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:03.089631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:34979)
0924 04:42:03.089945 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:03.089958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:03.090095 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:03.090382 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:03.090385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:03.090560 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:03.090666 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:03.090671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.091484 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.091487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:03.092079 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:03.092087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.092511 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.092514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:03.092575 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:03.093236 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:03.093247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:03.095214 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":83,"threads_started":1}
W20260924 04:42:03.095513 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:03.089685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34979 (local address 127.0.98.62:45427)
0924 04:42:03.089842 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:03.089847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:03.090070 (+ 223us) server_negotiation.cc:407] Connection header received
0924 04:42:03.090192 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:03.090195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:03.090240 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:03.090310 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:03.090792 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.091386 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.092193 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.092409 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.092488 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:03.095354 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34979: BlockingRecv error: recv got EOF from 127.0.98.2:34979 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:42:03 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:03 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:03.284119 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:03.278423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:44115)
0924 04:42:03.278711 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:03.278734 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:03.278861 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:03.279206 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:03.279208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:03.279429 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:03.279524 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:03.279529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.280339 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.280345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:03.280913 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:03.280920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.281390 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.281393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:03.281480 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:03.281995 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:03.282006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:03.284000 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
W20260924 04:42:03.284377 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:03.278509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44115 (local address 127.0.98.62:45427)
0924 04:42:03.278643 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:03.278648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:03.278815 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:42:03.278980 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:03.278984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:03.279055 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:03.279180 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:03.279653 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.280198 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.281030 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.281275 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.281385 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:03.284266 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44115: BlockingRecv error: recv got EOF from 127.0.98.3:44115 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 04:42:03 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:03 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:03.910630 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:03.904760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:50229)
0924 04:42:03.905141 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:03.905158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:03.905279 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:03.905625 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:03.905628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:03.905931 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:03.906127 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:03.906134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.906923 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.906926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:03.907583 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:03.907593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.908064 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.908067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:03.908137 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:03.908638 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:03.908651 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:03.910490 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
W20260924 04:42:03.910760 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:03.904851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50229 (local address 127.0.98.62:45427)
0924 04:42:03.905116 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:03.905120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:03.905237 (+ 117us) server_negotiation.cc:407] Connection header received
0924 04:42:03.905391 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:03.905394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:03.905452 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:03.905551 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:03.906281 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.906814 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.907698 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:03.907924 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:03.908005 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:03.910628 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50229: BlockingRecv error: recv got EOF from 127.0.98.1:50229 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
Sep 24 04:42:04 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:04 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:04.102649 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:04.096419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:42937)
0924 04:42:04.096706 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:04.096724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:04.096849 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:04.097283 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:04.097286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:04.097555 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:04.097659 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:04.097665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.098613 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.098616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:04.099353 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:04.099363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.099919 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.099922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:04.099988 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:04.100463 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:04.100475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:04.102534 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":101,"threads_started":1}
W20260924 04:42:04.102849 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:04.096585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42937 (local address 127.0.98.62:45427)
0924 04:42:04.096769 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:04.096776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:04.096794 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:42:04.097015 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:04.097020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:04.097080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:04.097177 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:04.097833 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.098512 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.099484 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.099804 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.099894 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:04.102690 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42937: BlockingRecv error: recv got EOF from 127.0.98.2:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 04:42:04 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:04 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:04.292024 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:04.285054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:40743)
0924 04:42:04.285387 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:04.285402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:04.285509 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:04.285971 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:04.285973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:04.286260 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:04.286359 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:04.286365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.287448 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.287453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:04.288297 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:04.288309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.289100 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.289107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:04.289219 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:04.289820 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:04.289832 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:04.291905 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":118,"threads_started":1}
W20260924 04:42:04.292301 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:04.285268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40743 (local address 127.0.98.62:45427)
0924 04:42:04.285416 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:04.285421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:04.285459 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:42:04.285649 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:04.285652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:04.285701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:04.285788 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:04.286515 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.287079 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.288472 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.288829 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.288952 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:04.292183 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40743: BlockingRecv error: recv got EOF from 127.0.98.3:40743 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 04:42:04 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:04 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:04.922997 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:04.911515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:46711)
0924 04:42:04.911786 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:04.911809 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:04.911900 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:04.912179 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:04.912181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:04.912404 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:04.912500 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:04.912506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.913336 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.913339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:04.913932 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:04.913942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.914442 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.914446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:04.914517 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:04.920956 (+ 6439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:04.920970 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:04.922871 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
W20260924 04:42:04.923319 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:04.911647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46711 (local address 127.0.98.62:45427)
0924 04:42:04.911945 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:04.911948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:04.911962 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:42:04.912000 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:04.912002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:04.912045 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:04.912149 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:04.912627 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.913202 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.914075 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:04.914306 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:04.914395 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:04.923127 (+ 8732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46711: BlockingRecv error: recv got EOF from 127.0.98.1:46711 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
Sep 24 04:42:05 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:05 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:05.109490 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:05.103475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:58903)
0924 04:42:05.103825 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:05.103837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:05.103982 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:05.104282 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:05.104285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:05.104494 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:05.104586 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:05.104592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.105458 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.105465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:05.106193 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:05.106204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.106696 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.106700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:05.106787 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:05.107403 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:05.107414 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:05.109363 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":90,"threads_started":1}
W20260924 04:42:05.109710 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:05.103579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58903 (local address 127.0.98.62:45427)
0924 04:42:05.103761 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:05.103767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:05.103917 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:42:05.104106 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:05.104109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:05.104165 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:05.104264 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:05.104748 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.105346 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.106323 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.106554 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.106664 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:05.109535 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58903: BlockingRecv error: recv got EOF from 127.0.98.2:58903 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 04:42:05 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:05 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:05.298840 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:05.292842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:47357)
0924 04:42:05.293140 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:05.293154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:05.293308 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:05.293528 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:05.293530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:05.293714 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:05.293810 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:05.293815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.294612 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.294614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:05.295332 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:05.295343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.295847 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.295851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:05.295931 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:05.296565 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:05.296576 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:05.298674 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":129,"threads_started":1}
W20260924 04:42:05.298951 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:05.292893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47357 (local address 127.0.98.62:45427)
0924 04:42:05.293056 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:05.293061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:05.293248 (+ 187us) server_negotiation.cc:407] Connection header received
0924 04:42:05.293381 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:05.293383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:05.293430 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:05.293502 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:05.293912 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.294493 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.295470 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.295714 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.295807 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:05.298830 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47357: BlockingRecv error: recv got EOF from 127.0.98.3:47357 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:42:05 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:05 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:05.930079 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:05.924074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:34611)
0924 04:42:05.924413 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:05.924428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:05.924530 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:05.924851 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:05.924854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:05.925033 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:05.925126 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:05.925132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.925892 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.925894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:05.926464 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:05.926472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.927076 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.927080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:05.927204 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:05.927818 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:05.927831 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:05.929964 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
W20260924 04:42:05.930269 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:05.924272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34611 (local address 127.0.98.62:45427)
0924 04:42:05.924476 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:05.924479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:05.924495 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:42:05.924650 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:05.924653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:05.924697 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:05.924779 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:05.925260 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.925796 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.926622 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:05.926916 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:05.927009 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:05.930100 (+ 3091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34611: BlockingRecv error: recv got EOF from 127.0.98.1:34611 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
Sep 24 04:42:06 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:42:06 dist-test-slave-npg2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:42:06.115933 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:06.110428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:50789)
0924 04:42:06.110710 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:06.110723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:06.110816 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:06.111052 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:06.111054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:06.111259 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:06.111388 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:06.111394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.112240 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.112243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:06.112832 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:06.112841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.113286 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.113289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:06.113354 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.113868 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:06.113880 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:06.115813 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45427: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
W20260924 04:42:06.116163 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:06.110487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50789 (local address 127.0.98.62:45427)
0924 04:42:06.110633 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:06.110636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:06.110794 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:42:06.110896 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:06.110898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:06.110939 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:06.111013 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:06.111526 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.112049 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.112943 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.113154 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.113274 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.115973 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50789: BlockingRecv error: recv got EOF from 127.0.98.2:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:42:06.189539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9137
I20260924 04:42:06.195088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9289
I20260924 04:42:06.200459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9422
I20260924 04:42:06.205888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9067
2026-09-24T04:42:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18130 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:42:06Z Disabled control of system clock
I20260924 04:42:06.236732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41067
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:42:06.349181 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:06.349471 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:06.349506 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:06.353569 9618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:42:06.353636 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:06.353650 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:06.353667 9618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:42:06.353683 9618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:42:06.358633 9618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:06.359747 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:06.361137 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:06.370078 9624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:06.370105 9626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:06.370174 9618 server_base.cc:1034] running on GCE node
W20260924 04:42:06.370011 9623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:06.371085 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:06.372323 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:06.373497 9618 hybrid_clock.cc:648] HybridClock initialized: now 1790224926373464 us; error 46 us; skew 500 ppm
I20260924 04:42:06.375569 9618 webserver.cc:460] Webserver started at http://127.0.98.62:35503/ using document root <none> and password file <none>
I20260924 04:42:06.376175 9618 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:06.376242 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:06.376459 9618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:06.378509 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "0f2650ed23424b169ef1c49303d7e1cd"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "7b33237ebbae746956577cec5c4cf6f9"
server_key_iv: "dae34811b32e549fdb114a971f11429f"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.379101 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f2650ed23424b169ef1c49303d7e1cd"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "7b33237ebbae746956577cec5c4cf6f9"
server_key_iv: "dae34811b32e549fdb114a971f11429f"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.382965 9618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:42:06.385702 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.386962 9618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:42:06.387063 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "0f2650ed23424b169ef1c49303d7e1cd"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "7b33237ebbae746956577cec5c4cf6f9"
server_key_iv: "dae34811b32e549fdb114a971f11429f"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.387285 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:06.423038 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:06.423938 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:06.424135 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:06.431355 9618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41067
I20260924 04:42:06.431365 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41067 every 8 connection(s)
I20260924 04:42:06.432417 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:42:06.433099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9618
I20260924 04:42:06.433228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:42:06.433565 392 external_mini_cluster.cc:1428] Setting key 5119095491845e437c7d56c67666dcd3
I20260924 04:42:06.436218 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:06.441797 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd: Bootstrap starting.
I20260924 04:42:06.443225 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:06.435162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.98.62:41067)
0924 04:42:06.435826 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:06.435842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:06.435878 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:42:06.436651 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:06.436671 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:06.437079 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:06.437431 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:06.437853 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.438820 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.439811 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.440289 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.440455 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.441383 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:06.441388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:06.441400 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:06.441814 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:06.441898 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:06.441927 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:06.442044 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:06.442164 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 04:42:06.442444 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260924 04:42:06.444324 9685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:06.445190 9685 log.cc:824] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:06.447188 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd: No bootstrap required, opened a new log
I20260924 04:42:06.451018 9685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2650ed23424b169ef1c49303d7e1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41067 } }
I20260924 04:42:06.451310 9685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:06.451351 9685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2650ed23424b169ef1c49303d7e1cd, State: Initialized, Role: FOLLOWER
I20260924 04:42:06.451727 9685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2650ed23424b169ef1c49303d7e1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41067 } }
I20260924 04:42:06.451838 9685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:42:06.451881 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:42:06.451952 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:06.452818 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2650ed23424b169ef1c49303d7e1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41067 } }
I20260924 04:42:06.453217 9685 leader_election.cc:302] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f2650ed23424b169ef1c49303d7e1cd; no voters:
I20260924 04:42:06.453505 9685 leader_election.cc:288] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:42:06.453634 9690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:42:06.453863 9690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [term 1 LEADER]: Becoming Leader. State: Replica: 0f2650ed23424b169ef1c49303d7e1cd, State: Running, Role: LEADER
I20260924 04:42:06.454187 9690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2650ed23424b169ef1c49303d7e1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41067 } }
I20260924 04:42:06.454891 9685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:42:06.455965 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f2650ed23424b169ef1c49303d7e1cd. Latest consensus state: current_term: 1 leader_uuid: "0f2650ed23424b169ef1c49303d7e1cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2650ed23424b169ef1c49303d7e1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41067 } } }
I20260924 04:42:06.456091 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:06.456447 9699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:42:06.456447 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f2650ed23424b169ef1c49303d7e1cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2650ed23424b169ef1c49303d7e1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41067 } } }
I20260924 04:42:06.456630 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:06.459656 9699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:42:06.465366 9699 catalog_manager.cc:1347] Generated new cluster ID: 4ba5f3cc687d4650a0cdc6332c604cf9
I20260924 04:42:06.465581 9699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:42:06.490183 9699 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:42:06.491441 9699 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:42:06.498091 9699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f2650ed23424b169ef1c49303d7e1cd: Generated new TSK 0
I20260924 04:42:06.498883 9699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:42:06.508406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41067
--builtin_ntp_servers=127.0.98.20:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:42:06.630321 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:06.630666 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:06.630712 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:06.638518 9709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:06.638623 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:06.638711 9709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:42:06.644482 9709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:06.645766 9709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:06.647341 9709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:06.658041 9717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:06.658042 9714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:06.658042 9715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:06.658160 9709 server_base.cc:1034] running on GCE node
I20260924 04:42:06.658747 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:06.659554 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:06.660789 9709 hybrid_clock.cc:648] HybridClock initialized: now 1790224926660770 us; error 30 us; skew 500 ppm
I20260924 04:42:06.662853 9709 webserver.cc:460] Webserver started at http://127.0.98.1:44343/ using document root <none> and password file <none>
I20260924 04:42:06.663563 9709 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:06.663667 9709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:06.663930 9709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:06.666198 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "cb8184bf2f32468995e83b8eb82fdd95"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "6701201acd5328cdd5d55eeb8f1b4bf4"
server_key_iv: "9e7e40ad976e91a1370be2be0a5a9dc3"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.666824 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cb8184bf2f32468995e83b8eb82fdd95"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "6701201acd5328cdd5d55eeb8f1b4bf4"
server_key_iv: "9e7e40ad976e91a1370be2be0a5a9dc3"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.670874 9709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 04:42:06.673707 9723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.675024 9709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:42:06.675208 9709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "cb8184bf2f32468995e83b8eb82fdd95"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "6701201acd5328cdd5d55eeb8f1b4bf4"
server_key_iv: "9e7e40ad976e91a1370be2be0a5a9dc3"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.675333 9709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:06.690553 9709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:06.691473 9709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:06.691692 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:06.692412 9709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:06.693390 9709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:06.693464 9709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.693532 9709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:06.693583 9709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.702659 9709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37147
I20260924 04:42:06.702679 9836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37147 every 8 connection(s)
I20260924 04:42:06.703696 9709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:42:06.705966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9709
I20260924 04:42:06.706147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:42:06.706465 392 external_mini_cluster.cc:1428] Setting key 4d2b0a30e77902e7ffff74c1a53161de
I20260924 04:42:06.708470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41067
--builtin_ntp_servers=127.0.98.20:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:42:06.714983 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:06.705610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43271 (local address 127.0.98.62:41067)
0924 04:42:06.705842 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:06.705848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:06.707008 (+ 1160us) server_negotiation.cc:407] Connection header received
0924 04:42:06.708588 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:06.708594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:06.708668 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:06.708764 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:06.709954 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.710828 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.712741 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.712966 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.713058 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.714471 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:06.714474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:06.714476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:06.714489 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:06.714508 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:06.714516 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:06.714578 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:06.714792 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:42:06.714863 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 04:42:06.716379 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:06.706035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41067 (local address 127.0.98.1:43271)
0924 04:42:06.706824 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:06.706878 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:06.708365 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:06.708995 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:06.709005 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:06.709116 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:06.709786 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:06.709803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.710985 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.710992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:06.712598 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:06.712613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.713150 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.713157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:06.713409 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.714115 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:42:06.714119 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:06.714186 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:42:06.714286 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:06.714614 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:06.714625 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:06.714635 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 04:42:06.714795 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:42:06.715184 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":169,"threads_started":1}
I20260924 04:42:06.717785 9837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41067
I20260924 04:42:06.718035 9837 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:06.718657 9837 heartbeater.cc:511] Master 127.0.98.62:41067 requested a full tablet report, sending...
I20260924 04:42:06.720278 9649 ts_manager.cc:194] Registered new tserver with Master: cb8184bf2f32468995e83b8eb82fdd95 (127.0.98.1:37147)
I20260924 04:42:06.721443 9649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43271
W20260924 04:42:06.823976 9840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:06.824302 9840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:06.824376 9840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:06.828608 9840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:06.828709 9840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:06.828804 9840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:42:06.833554 9840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:06.834717 9840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:06.836151 9840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:06.846350 9845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:06.846370 9846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:06.846408 9840 server_base.cc:1034] running on GCE node
W20260924 04:42:06.846524 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:06.847062 9840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:06.847781 9840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:06.848973 9840 hybrid_clock.cc:648] HybridClock initialized: now 1790224926848948 us; error 37 us; skew 500 ppm
I20260924 04:42:06.850983 9840 webserver.cc:460] Webserver started at http://127.0.98.2:36547/ using document root <none> and password file <none>
I20260924 04:42:06.851647 9840 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:06.851749 9840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:06.851991 9840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:06.854280 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "6597e8167ec249db8cf3ae77e0895af2"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "c04c0bea11a43091e9b550a2a3bba2ce"
server_key_iv: "8eff40d151f36da2d69f661010d3829a"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.854887 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6597e8167ec249db8cf3ae77e0895af2"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "c04c0bea11a43091e9b550a2a3bba2ce"
server_key_iv: "8eff40d151f36da2d69f661010d3829a"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.859097 9840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:42:06.861876 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.863268 9840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:42:06.863407 9840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "6597e8167ec249db8cf3ae77e0895af2"
format_stamp: "Formatted at 2026-09-24 04:42:06 on dist-test-slave-npg2"
server_key: "c04c0bea11a43091e9b550a2a3bba2ce"
server_key_iv: "8eff40d151f36da2d69f661010d3829a"
server_key_version: "encryptionkey@0"
I20260924 04:42:06.863530 9840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:06.897065 9840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:06.897859 9840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:06.898068 9840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:06.898670 9840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:06.899744 9840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:06.899816 9840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.899883 9840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:06.899925 9840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:06.909488 9840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44839
I20260924 04:42:06.909505 9967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44839 every 8 connection(s)
I20260924 04:42:06.910476 9840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:42:06.916399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9840
I20260924 04:42:06.916561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:42:06.916906 392 external_mini_cluster.cc:1428] Setting key ea6621c03b8e1abbc39f7a88899188e4
I20260924 04:42:06.919049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41067
--builtin_ntp_servers=127.0.98.20:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:42:06.920881 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:06.912439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54757 (local address 127.0.98.62:41067)
0924 04:42:06.912604 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:06.912609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:06.913763 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 04:42:06.915328 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:06.915331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:06.915390 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:06.915477 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:06.916519 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.917292 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.918567 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.918907 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.919024 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.920357 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:06.920361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:06.920363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:06.920377 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:06.920403 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:06.920412 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:06.920489 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:06.920656 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:42:06.920717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:42:06.921844 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:06.912907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41067 (local address 127.0.98.2:54757)
0924 04:42:06.913607 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:06.913649 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:06.914920 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:06.915670 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:06.915678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:06.915778 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:06.916340 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:06.916354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.917430 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.917435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:06.918448 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:06.918457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:06.919118 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:06.919121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:06.919398 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:06.920075 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:42:06.920076 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:06.920119 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:42:06.920187 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:06.920502 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:06.920513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:06.920524 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 04:42:06.920646 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:42:06.920951 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260924 04:42:06.923112 9968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41067
I20260924 04:42:06.923460 9968 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:06.924014 9968 heartbeater.cc:511] Master 127.0.98.62:41067 requested a full tablet report, sending...
I20260924 04:42:06.925230 9649 ts_manager.cc:194] Registered new tserver with Master: 6597e8167ec249db8cf3ae77e0895af2 (127.0.98.2:44839)
I20260924 04:42:06.926136 9649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54757
W20260924 04:42:07.035660 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:07.035940 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:07.035979 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:07.040266 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:07.040339 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:07.040405 9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:42:07.045471 9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:07.046622 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:07.048174 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:07.058403 9979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.058454 9976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.058403 9977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.058630 9971 server_base.cc:1034] running on GCE node
I20260924 04:42:07.059366 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:07.060087 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:07.061347 9971 hybrid_clock.cc:648] HybridClock initialized: now 1790224927061331 us; error 31 us; skew 500 ppm
I20260924 04:42:07.063654 9971 webserver.cc:460] Webserver started at http://127.0.98.3:40899/ using document root <none> and password file <none>
I20260924 04:42:07.064280 9971 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:07.064350 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:07.064587 9971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:07.068745 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "993fd9ee3f8f4a97a18916c78d66b583"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "23d8a0524e1d54543d6121f9fb6e4584"
server_key_iv: "f3ca023ccc1b6207d2a10472d3085a5b"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.069335 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "993fd9ee3f8f4a97a18916c78d66b583"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "23d8a0524e1d54543d6121f9fb6e4584"
server_key_iv: "f3ca023ccc1b6207d2a10472d3085a5b"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.073300 9971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:42:07.076228 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.077543 9971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:42:07.077697 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "993fd9ee3f8f4a97a18916c78d66b583"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "23d8a0524e1d54543d6121f9fb6e4584"
server_key_iv: "f3ca023ccc1b6207d2a10472d3085a5b"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.077829 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:07.097302 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:07.098067 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:07.098292 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:07.098959 9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:07.100003 9971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:07.100049 9971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.100090 9971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:07.100104 9971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.109100 9971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44055
I20260924 04:42:07.109126 10098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44055 every 8 connection(s)
I20260924 04:42:07.110033 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:42:07.116011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 9971
I20260924 04:42:07.116112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:42:07.116376 392 external_mini_cluster.cc:1428] Setting key 09f28a7864377e7e174b0bd3d1446fae
I20260924 04:42:07.119966 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.112063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53213 (local address 127.0.98.62:41067)
0924 04:42:07.112232 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.112236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.113381 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 04:42:07.114718 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.114721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.114771 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.114851 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.115855 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:07.116493 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:07.117951 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:07.118175 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:07.118290 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:07.119470 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:07.119474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:07.119477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:07.119497 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:07.119515 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:07.119523 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:07.119599 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:07.119766 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:42:07.119826 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:42:07.120820 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.112535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41067 (local address 127.0.98.3:53213)
0924 04:42:07.113230 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:07.113270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:07.114520 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:07.115021 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:07.115029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:07.115111 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:07.115694 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:07.115706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:07.116619 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:07.116623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:07.117830 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:07.117839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:07.118326 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:07.118330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:07.118502 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:07.119207 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:42:07.119209 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:07.119252 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:42:07.119316 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:07.119612 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:07.119623 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:07.119629 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:42:07.119766 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 04:42:07.120049 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260924 04:42:07.121892 10099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41067
I20260924 04:42:07.122138 10099 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:07.122665 10099 heartbeater.cc:511] Master 127.0.98.62:41067 requested a full tablet report, sending...
I20260924 04:42:07.123718 9649 ts_manager.cc:194] Registered new tserver with Master: 993fd9ee3f8f4a97a18916c78d66b583 (127.0.98.3:44055)
I20260924 04:42:07.124331 9649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53213
I20260924 04:42:07.130442 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:42:07.133445 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:07.132371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.0.98.62:41067)
0924 04:42:07.132527 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.132533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.132729 (+ 196us) server_negotiation.cc:407] Connection header received
0924 04:42:07.132898 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.132901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.132953 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.133035 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.133286 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33418: BlockingRecv error: recv got EOF from 127.0.0.1:33418 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:42:07.133603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9709
I20260924 04:42:07.139180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9840
I20260924 04:42:07.144738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9971
I20260924 04:42:07.149945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 9618
2026-09-24T04:42:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (943 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:42:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:42:07Z Disabled control of system clock
I20260924 04:42:07.180022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:42:07.296326 10110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:07.296633 10110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:07.296674 10110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:07.301162 10110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:42:07.301255 10110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:07.301271 10110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:07.301290 10110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:42:07.301306 10110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:42:07.306484 10110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:07.307672 10110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:07.309077 10110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:07.317850 10116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.317819 10118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.317852 10115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.318341 10110 server_base.cc:1034] running on GCE node
I20260924 04:42:07.318920 10110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:07.320305 10110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:07.321508 10110 hybrid_clock.cc:648] HybridClock initialized: now 1790224927321491 us; error 37 us; skew 500 ppm
I20260924 04:42:07.323643 10110 webserver.cc:460] Webserver started at http://127.0.98.62:35367/ using document root <none> and password file <none>
I20260924 04:42:07.324321 10110 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:07.324426 10110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:07.324669 10110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:07.326960 10110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "b8541700394f405786e53a42109961ff"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "2ac91d0157d480c378167b5a4283dcc9"
server_key_iv: "624f1484edf330ed4ee3eb2b398c7e79"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.327654 10110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8541700394f405786e53a42109961ff"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "2ac91d0157d480c378167b5a4283dcc9"
server_key_iv: "624f1484edf330ed4ee3eb2b398c7e79"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.331656 10110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:42:07.334256 10124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.335741 10110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:42:07.335902 10110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "b8541700394f405786e53a42109961ff"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "2ac91d0157d480c378167b5a4283dcc9"
server_key_iv: "624f1484edf330ed4ee3eb2b398c7e79"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.336031 10110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:07.340910 10110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:07.341554 10110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:07.341755 10110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:07.348759 10110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33129
I20260924 04:42:07.348807 10176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33129 every 8 connection(s)
I20260924 04:42:07.349763 10110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:42:07.353264 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:07.356244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10110
I20260924 04:42:07.356398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:42:07.356693 392 external_mini_cluster.cc:1428] Setting key 00e3372b7dfeaae9523c517068a9f6e3
I20260924 04:42:07.359397 10177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff: Bootstrap starting.
I20260924 04:42:07.361878 10177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:07.362566 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.358099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.62:33129)
0924 04:42:07.358784 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.358800 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.358837 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:42:07.359599 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.359621 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.359988 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.360345 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.361158 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:07.361163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:07.361175 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:07.361186 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:07.361302 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:07.361323 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:07.361446 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:07.361547 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 04:42:07.361794 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":133,"threads_started":1}
I20260924 04:42:07.362708 10177 log.cc:824] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:07.365024 10177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff: No bootstrap required, opened a new log
I20260924 04:42:07.368824 10177 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8541700394f405786e53a42109961ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33129 } }
I20260924 04:42:07.369099 10177 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:07.369163 10177 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8541700394f405786e53a42109961ff, State: Initialized, Role: FOLLOWER
I20260924 04:42:07.369587 10177 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8541700394f405786e53a42109961ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33129 } }
I20260924 04:42:07.369750 10177 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:42:07.369833 10177 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:42:07.369954 10177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:07.371083 10177 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8541700394f405786e53a42109961ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33129 } }
I20260924 04:42:07.371469 10177 leader_election.cc:302] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8541700394f405786e53a42109961ff; no voters:
I20260924 04:42:07.371789 10177 leader_election.cc:288] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:42:07.372072 10182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:42:07.372314 10182 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [term 1 LEADER]: Becoming Leader. State: Replica: b8541700394f405786e53a42109961ff, State: Running, Role: LEADER
I20260924 04:42:07.372653 10182 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8541700394f405786e53a42109961ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33129 } }
I20260924 04:42:07.373116 10177 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:42:07.374486 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8541700394f405786e53a42109961ff. Latest consensus state: current_term: 1 leader_uuid: "b8541700394f405786e53a42109961ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8541700394f405786e53a42109961ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33129 } } }
I20260924 04:42:07.374636 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:07.374454 10183 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8541700394f405786e53a42109961ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8541700394f405786e53a42109961ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33129 } } }
I20260924 04:42:07.374718 10183 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:07.375219 10194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:42:07.378432 10194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:42:07.384438 10194 catalog_manager.cc:1347] Generated new cluster ID: 28a7eefce3634e91b3a41696dea2780b
I20260924 04:42:07.384543 10194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:42:07.402247 10194 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:42:07.402675 10194 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:42:07.414052 10194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8541700394f405786e53a42109961ff: Generated new TSK 0
I20260924 04:42:07.414906 10194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:42:07.420179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33129
--builtin_ntp_servers=127.0.98.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:42:07.540432 10201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:07.540751 10201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:07.540831 10201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:07.544955 10201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:07.545044 10201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:07.545141 10201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:42:07.550424 10201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:07.551851 10201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:07.553458 10201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:07.563879 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.563880 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.564028 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.564226 10201 server_base.cc:1034] running on GCE node
I20260924 04:42:07.564812 10201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:07.565474 10201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:07.566676 10201 hybrid_clock.cc:648] HybridClock initialized: now 1790224927566663 us; error 37 us; skew 500 ppm
I20260924 04:42:07.568809 10201 webserver.cc:460] Webserver started at http://127.0.98.1:41549/ using document root <none> and password file <none>
I20260924 04:42:07.569416 10201 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:07.569507 10201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:07.569729 10201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:07.572031 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "990df7a8ee8945779137dd98d15ecfed"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "4be2cccf19f368bf3995adb935952965"
server_key_iv: "9998ee3643efddd3d72cf2caea2af520"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.572616 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "990df7a8ee8945779137dd98d15ecfed"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "4be2cccf19f368bf3995adb935952965"
server_key_iv: "9998ee3643efddd3d72cf2caea2af520"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.576853 10201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:42:07.579504 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.580911 10201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:42:07.581056 10201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "990df7a8ee8945779137dd98d15ecfed"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "4be2cccf19f368bf3995adb935952965"
server_key_iv: "9998ee3643efddd3d72cf2caea2af520"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.581184 10201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:07.586150 10201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:07.586823 10201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:07.587025 10201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:07.587667 10201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:07.588653 10201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:07.588723 10201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.588788 10201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:07.588829 10201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.598176 10201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43855
I20260924 04:42:07.598264 10328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43855 every 8 connection(s)
I20260924 04:42:07.599254 10201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:42:07.605762 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.601262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41437 (local address 127.0.98.62:33129)
0924 04:42:07.601431 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.601436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.602510 (+ 1074us) server_negotiation.cc:407] Connection header received
0924 04:42:07.603977 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.603980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.604019 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.604099 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.605370 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:07.605372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:07.605375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:07.605385 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:07.605401 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:07.605408 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:07.605484 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:07.605616 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:42:07.605653 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:42:07.606704 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.601641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33129 (local address 127.0.98.1:41437)
0924 04:42:07.602360 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:07.602401 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:07.603770 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:07.604381 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:07.604392 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:07.604485 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:07.605124 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:42:07.605126 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:07.605167 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:42:07.605242 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:07.605495 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:07.605501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:07.605505 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:42:07.605637 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 04:42:07.605846 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260924 04:42:07.607230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10201
I20260924 04:42:07.607370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:42:07.607672 392 external_mini_cluster.cc:1428] Setting key 61c8e6e533d9429513bf87931fbf034f
I20260924 04:42:07.608021 10329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33129
I20260924 04:42:07.608283 10329 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:07.608686 10329 heartbeater.cc:511] Master 127.0.98.62:33129 requested a full tablet report, sending...
I20260924 04:42:07.609653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33129
--builtin_ntp_servers=127.0.98.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:42:07.610375 10141 ts_manager.cc:194] Registered new tserver with Master: 990df7a8ee8945779137dd98d15ecfed (127.0.98.1:43855)
W20260924 04:42:07.725499 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:07.725782 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:07.725823 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:07.730047 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:07.730110 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:07.730170 10332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:42:07.734932 10332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:07.736128 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:07.737454 10332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:07.747200 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.747270 10340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.747251 10332 server_base.cc:1034] running on GCE node
W20260924 04:42:07.747201 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.747884 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:07.748481 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:07.749648 10332 hybrid_clock.cc:648] HybridClock initialized: now 1790224927749634 us; error 34 us; skew 500 ppm
I20260924 04:42:07.751729 10332 webserver.cc:460] Webserver started at http://127.0.98.2:38341/ using document root <none> and password file <none>
I20260924 04:42:07.752321 10332 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:07.752383 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:07.752592 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:07.754609 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "d56aabdc36ed41bba8921b3207ffd287"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "d21f480f85dbddc9aaeed76eb8b572ba"
server_key_iv: "a6ee9fa0087421ad1a215ede9580e135"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.755230 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d56aabdc36ed41bba8921b3207ffd287"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "d21f480f85dbddc9aaeed76eb8b572ba"
server_key_iv: "a6ee9fa0087421ad1a215ede9580e135"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.758945 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:42:07.761411 10346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.762701 10332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:42:07.762836 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "d56aabdc36ed41bba8921b3207ffd287"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "d21f480f85dbddc9aaeed76eb8b572ba"
server_key_iv: "a6ee9fa0087421ad1a215ede9580e135"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.762951 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:07.767690 10332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:07.768308 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:07.768494 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:07.769038 10332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:07.769912 10332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:07.769982 10332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.770052 10332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:07.770103 10332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.779515 10332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35619
I20260924 04:42:07.779570 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35619 every 8 connection(s)
I20260924 04:42:07.780509 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:42:07.786736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10332
I20260924 04:42:07.786895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:42:07.787194 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.782393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48761 (local address 127.0.98.62:33129)
0924 04:42:07.782531 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.782535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.783749 (+ 1214us) server_negotiation.cc:407] Connection header received
0924 04:42:07.785249 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.785252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.785301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.785422 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.786610 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:07.786613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:07.786616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:07.786630 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:07.786658 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:07.786668 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:07.786775 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:07.786965 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 04:42:07.787017 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:42:07.787250 392 external_mini_cluster.cc:1428] Setting key f8356225aff1f7e380c4fd44929f5890
I20260924 04:42:07.788277 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.782813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33129 (local address 127.0.98.2:48761)
0924 04:42:07.783586 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:07.783629 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:07.785007 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:07.785596 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:07.785602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:07.785703 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:07.786334 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:42:07.786336 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:07.786380 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:42:07.786450 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:07.786754 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:07.786763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:07.786768 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:42:07.786987 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 04:42:07.787280 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260924 04:42:07.789279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33129
--builtin_ntp_servers=127.0.98.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:42:07.789481 10460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33129
I20260924 04:42:07.789731 10460 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:07.790132 10460 heartbeater.cc:511] Master 127.0.98.62:33129 requested a full tablet report, sending...
I20260924 04:42:07.791232 10141 ts_manager.cc:194] Registered new tserver with Master: d56aabdc36ed41bba8921b3207ffd287 (127.0.98.2:35619)
W20260924 04:42:07.903404 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:07.903666 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:07.903707 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:07.907823 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:07.907881 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:07.907965 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:42:07.912734 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:07.913915 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:07.915557 10463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:07.925412 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.925434 10463 server_base.cc:1034] running on GCE node
W20260924 04:42:07.925426 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:07.925441 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:07.926144 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:07.926842 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:07.928056 10463 hybrid_clock.cc:648] HybridClock initialized: now 1790224927928022 us; error 82 us; skew 500 ppm
I20260924 04:42:07.930052 10463 webserver.cc:460] Webserver started at http://127.0.98.3:42067/ using document root <none> and password file <none>
I20260924 04:42:07.930750 10463 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:07.930842 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:07.931061 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:07.933176 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "1244fcd9e828408aae3871f240420f75"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "f2fb0ad9da3edac01f710726bc8a752c"
server_key_iv: "261374a7e4bddaa915bd34b741bfe3f7"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.933753 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1244fcd9e828408aae3871f240420f75"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "f2fb0ad9da3edac01f710726bc8a752c"
server_key_iv: "261374a7e4bddaa915bd34b741bfe3f7"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.937628 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:42:07.940189 10477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.941519 10463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:42:07.941653 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "1244fcd9e828408aae3871f240420f75"
format_stamp: "Formatted at 2026-09-24 04:42:07 on dist-test-slave-npg2"
server_key: "f2fb0ad9da3edac01f710726bc8a752c"
server_key_iv: "261374a7e4bddaa915bd34b741bfe3f7"
server_key_version: "encryptionkey@0"
I20260924 04:42:07.941773 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:07.946478 10463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:07.947082 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:07.947304 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:07.947841 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:07.948722 10463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:07.948800 10463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.948890 10463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:07.948943 10463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:07.957974 10463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33419
I20260924 04:42:07.957995 10590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33419 every 8 connection(s)
I20260924 04:42:07.958877 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:42:07.965706 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.960879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37839 (local address 127.0.98.62:33129)
0924 04:42:07.961049 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.961055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.962377 (+ 1322us) server_negotiation.cc:407] Connection header received
0924 04:42:07.963897 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.963900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.963939 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.964017 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.965264 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:07.965267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:07.965269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:42:07.965280 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:07.965297 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:42:07.965304 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:07.965383 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:07.965557 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 04:42:07.965596 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:42:07.965919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10463
I20260924 04:42:07.966037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:42:07.966390 392 external_mini_cluster.cc:1428] Setting key d8d120f3f014f0ea355b2d0c96a05f06
I20260924 04:42:07.967197 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:07.961278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33129 (local address 127.0.98.3:37839)
0924 04:42:07.962186 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:07.962251 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:07.963651 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:07.964256 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:07.964263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:07.964355 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:07.964984 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:42:07.964985 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:07.965032 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:42:07.965107 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:07.965430 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:07.965436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:07.965440 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:42:07.965588 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:42:07.965958 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":104,"threads_started":1}
I20260924 04:42:07.968420 10591 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33129
I20260924 04:42:07.968680 10591 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:07.969106 10591 heartbeater.cc:511] Master 127.0.98.62:33129 requested a full tablet report, sending...
I20260924 04:42:07.970191 10137 ts_manager.cc:194] Registered new tserver with Master: 1244fcd9e828408aae3871f240420f75 (127.0.98.3:33419)
I20260924 04:42:07.981246 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:42:07.984548 10180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:42:07.983568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.0.98.62:33129)
0924 04:42:07.983727 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:07.983731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:07.983932 (+ 201us) server_negotiation.cc:407] Connection header received
0924 04:42:07.984063 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:07.984066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:07.984111 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:07.984175 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:07.984412 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55664: BlockingRecv error: recv got EOF from 127.0.0.1:55664 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 04:42:07.984827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10201
I20260924 04:42:07.990746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10332
I20260924 04:42:07.996394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10463
I20260924 04:42:08.001883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10110
2026-09-24T04:42:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (852 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:42:08 dist-test-slave-npg2 krb5kdc[10601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:42:08 dist-test-slave-npg2 krb5kdc[10601](info): set up 2 sockets
Sep 24 04:42:08 dist-test-slave-npg2 krb5kdc[10601](info): commencing operation
krb5kdc: starting...
W20260924 04:42:09.044118 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:42:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:42:09.178618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:42:09.291909 10614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:09.292196 10614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:09.292237 10614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:09.296409 10614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:42:09.296476 10614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:09.296496 10614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:09.296514 10614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:42:09.296528 10614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:42:09.301982 10614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:09.303078 10614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:09.304133 10614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:09.312969 10620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:09.312999 10619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:09.312963 10614 server_base.cc:1034] running on GCE node
W20260924 04:42:09.313004 10622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:09.313903 10614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:09.315256 10614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:09.316462 10614 hybrid_clock.cc:648] HybridClock initialized: now 1790224929316434 us; error 49 us; skew 500 ppm
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:09.319571 10614 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:42:09.320588 10614 webserver.cc:460] Webserver started at http://127.0.98.62:46779/ using document root <none> and password file <none>
I20260924 04:42:09.321200 10614 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:09.321302 10614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:09.321556 10614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:09.324028 10614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "4dbf8da0ea4a4f92a43d507ae1c21599"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "769b9f8dcf7ed02dc10bdbda6bbd1c10"
server_key_iv: "78d51ba4e143826cdf6eb7c68848f7d0"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.324661 10614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "4dbf8da0ea4a4f92a43d507ae1c21599"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "769b9f8dcf7ed02dc10bdbda6bbd1c10"
server_key_iv: "78d51ba4e143826cdf6eb7c68848f7d0"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.328763 10614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:42:09.331559 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.332921 10614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:42:09.333072 10614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "4dbf8da0ea4a4f92a43d507ae1c21599"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "769b9f8dcf7ed02dc10bdbda6bbd1c10"
server_key_iv: "78d51ba4e143826cdf6eb7c68848f7d0"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.333190 10614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:09.356922 10614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:09.360289 10614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:09.360518 10614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:09.368046 10614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42511
I20260924 04:42:09.368419 10681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42511 every 8 connection(s)
I20260924 04:42:09.369058 10614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:42:09.372076 10682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:09.376003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10614
I20260924 04:42:09.376147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:42:09.376504 392 external_mini_cluster.cc:1428] Setting key 5cb1b5a7e554fa07eb21f1f04197363a
I20260924 04:42:09.378511 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: Bootstrap starting.
I20260924 04:42:09.381035 10682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:09.381841 10682 log.cc:824] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:09.384092 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: No bootstrap required, opened a new log
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:09.387727 10682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260924 04:42:09.388048 10682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:09.388135 10682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbf8da0ea4a4f92a43d507ae1c21599, State: Initialized, Role: FOLLOWER
I20260924 04:42:09.388633 10682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260924 04:42:09.388798 10682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:42:09.388878 10682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:42:09.388995 10682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:09.390182 10682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260924 04:42:09.390549 10682 leader_election.cc:302] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dbf8da0ea4a4f92a43d507ae1c21599; no voters:
I20260924 04:42:09.390902 10682 leader_election.cc:288] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:42:09.390975 10687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:42:09.391222 10687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [term 1 LEADER]: Becoming Leader. State: Replica: 4dbf8da0ea4a4f92a43d507ae1c21599, State: Running, Role: LEADER
I20260924 04:42:09.391546 10687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260924 04:42:09.392341 10682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:42:09.392774 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:09.377959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.0.98.62:42511)
0924 04:42:09.378409 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:09.378423 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:09.378458 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:42:09.379167 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:09.379199 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:09.379533 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:09.379869 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:09.380662 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.381727 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.382641 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.383162 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.383351 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:09.385913 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:09.385918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:09.385931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:09.385957 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:09.388720 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:09.389327 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:09.389332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:09.389337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:09.389446 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:09.389792 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:09.389796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:09.389797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:09.390294 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:09.390515 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:09.390931 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 04:42:09.391747 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260924 04:42:09.393743 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260924 04:42:09.393770 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dbf8da0ea4a4f92a43d507ae1c21599. Latest consensus state: current_term: 1 leader_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbf8da0ea4a4f92a43d507ae1c21599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260924 04:42:09.393877 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:09.393885 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599 [sys.catalog]: This master's current role is: LEADER
W20260924 04:42:09.397863 10702 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:42:09.397953 10702 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:42:09.398078 10696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:42:09.398885 10696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:42:09.405224 10696 catalog_manager.cc:1347] Generated new cluster ID: 5804723de25d432a8441304a9a9c2e02
I20260924 04:42:09.405331 10696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:42:09.433604 10696 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:42:09.434624 10696 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:42:09.444552 10696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: Generated new TSK 0
I20260924 04:42:09.445384 10696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:42:09.483260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:42:09.599705 10708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:09.599996 10708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:09.600040 10708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:09.604173 10708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:09.604233 10708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:09.604322 10708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:42:09.609700 10708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:09.610839 10708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:09.611967 10708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:09.621798 10716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:09.621801 10713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:09.621950 10708 server_base.cc:1034] running on GCE node
W20260924 04:42:09.621798 10714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:09.622493 10708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:09.623241 10708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:09.624459 10708 hybrid_clock.cc:648] HybridClock initialized: now 1790224929624441 us; error 37 us; skew 500 ppm
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:09.627545 10708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:42:09.628489 10708 webserver.cc:460] Webserver started at http://127.0.98.1:40685/ using document root <none> and password file <none>
I20260924 04:42:09.629108 10708 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:09.629209 10708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:09.629439 10708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:09.631696 10708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "6caf471898244123be9d6a5f97efad6f"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "bf09c63db3e1c178e673c48688948e20"
server_key_iv: "112b8ec95403813210608f821cfd45f9"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.632268 10708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6caf471898244123be9d6a5f97efad6f"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "bf09c63db3e1c178e673c48688948e20"
server_key_iv: "112b8ec95403813210608f821cfd45f9"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.636466 10708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:42:09.639565 10723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.641012 10708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:42:09.641175 10708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "6caf471898244123be9d6a5f97efad6f"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "bf09c63db3e1c178e673c48688948e20"
server_key_iv: "112b8ec95403813210608f821cfd45f9"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.641377 10708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:09.659798 10708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:09.663101 10708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:09.663371 10708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:09.663973 10708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:09.664803 10708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:09.664870 10708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.664937 10708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:09.664984 10708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.674383 10708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45241
I20260924 04:42:09.674431 10836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45241 every 8 connection(s)
I20260924 04:42:09.675433 10708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:42:09.680078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10708
I20260924 04:42:09.680231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:42:09.680540 392 external_mini_cluster.cc:1428] Setting key 9523ec1799cbeb52cc59eeaca2bea40a
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:42:09.691077 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:09.677440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33439 (local address 127.0.98.62:42511)
0924 04:42:09.677598 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:09.677601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:09.678522 (+ 921us) server_negotiation.cc:407] Connection header received
0924 04:42:09.679978 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:09.679981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:09.680030 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:09.680110 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:09.681695 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.682350 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.683686 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.683942 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.684032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:09.687189 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:09.687193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:09.687195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:09.687221 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:09.688813 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:09.689379 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:09.689382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:09.689384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:09.689466 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:09.689817 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:09.689822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:09.689824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:09.690023 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:09.690134 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:09.690806 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 04:42:09.690946 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:42:09.691984 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:09.677848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:33439)
0924 04:42:09.678362 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:09.678407 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:09.679706 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:09.680343 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:09.680353 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:09.680876 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:09.681543 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:09.681560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.682487 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.682497 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:09.683521 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:09.683536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.684110 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.684116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:09.684356 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:09.685077 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:09.685106 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:09.686932 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:09.688949 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:09.688956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:09.688980 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:09.689263 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:09.689564 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:09.689569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:09.689571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:09.689721 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:09.690132 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:09.690139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:09.690485 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 04:42:09.690699 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 04:42:09.690984 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:42:09.693578 10837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260924 04:42:09.693846 10837 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:09.694447 10837 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260924 04:42:09.696269 10644 ts_manager.cc:194] Registered new tserver with Master: 6caf471898244123be9d6a5f97efad6f (127.0.98.1:45241)
I20260924 04:42:09.697508 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:42:09.711594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:42:09.824697 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:09.824975 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:09.825021 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:09.829200 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:09.829269 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:09.829370 10842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:42:09.835253 10842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:09.836452 10842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:09.837661 10842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:42:09.848460 10842 server_base.cc:1034] running on GCE node
W20260924 04:42:09.848379 10847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:09.848383 10850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:09.848402 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:09.849044 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:09.849753 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:09.850936 10842 hybrid_clock.cc:648] HybridClock initialized: now 1790224929850908 us; error 41 us; skew 500 ppm
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:09.853938 10842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:42:09.854861 10842 webserver.cc:460] Webserver started at http://127.0.98.2:43241/ using document root <none> and password file <none>
I20260924 04:42:09.855517 10842 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:09.855607 10842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:09.855841 10842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:09.857985 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "fefce902cdcc4e2d92cf1666038fed57"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "11f2d182dbc3033bb854d236cd926074"
server_key_iv: "bc2c886bac9c27eab7bfa3832b8d3035"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.858527 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fefce902cdcc4e2d92cf1666038fed57"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "11f2d182dbc3033bb854d236cd926074"
server_key_iv: "bc2c886bac9c27eab7bfa3832b8d3035"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.862309 10842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:42:09.864872 10857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.866119 10842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 04:42:09.866230 10842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "fefce902cdcc4e2d92cf1666038fed57"
format_stamp: "Formatted at 2026-09-24 04:42:09 on dist-test-slave-npg2"
server_key: "11f2d182dbc3033bb854d236cd926074"
server_key_iv: "bc2c886bac9c27eab7bfa3832b8d3035"
server_key_version: "encryptionkey@0"
I20260924 04:42:09.866384 10842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:09.896817 10842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:09.900384 10842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:09.900592 10842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:09.901201 10842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:09.902033 10842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:09.902103 10842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.902168 10842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:09.902199 10842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:09.910967 10842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38673
I20260924 04:42:09.911003 10970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38673 every 8 connection(s)
I20260924 04:42:09.911968 10842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:42:09.918469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10842
I20260924 04:42:09.918574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:42:09.918905 392 external_mini_cluster.cc:1428] Setting key 3bd8fba8f1e92911927ef81ce7b84a5e
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 04:42:09 dist-test-slave-npg2 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:09.936151 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:09.914346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38941 (local address 127.0.98.62:42511)
0924 04:42:09.914480 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:09.914484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:09.915367 (+ 883us) server_negotiation.cc:407] Connection header received
0924 04:42:09.916722 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:09.916726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:09.916826 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:09.916928 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:09.918197 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.918791 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.920121 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.920349 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.920472 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:09.932035 (+ 11563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:09.932039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:09.932041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:09.932073 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:09.933790 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:09.934388 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:09.934394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:09.934401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:09.934482 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:09.934859 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:09.934863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:09.934864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:09.935068 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:09.935281 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:09.935846 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 04:42:09.935979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:42:09.937173 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:09.914709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:38941)
0924 04:42:09.915213 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:09.915256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:09.916486 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:09.917114 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:09.917121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:09.917533 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:09.918041 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:09.918054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.918918 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.918921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:09.919989 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:09.919999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:09.920512 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:09.920516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:09.920693 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:09.927622 (+ 6929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:09.927643 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:09.931756 (+ 4113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:09.933921 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:09.933928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:09.933945 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:09.934220 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:09.934587 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:09.934592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:09.934599 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:09.934749 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:09.935286 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:09.935295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:09.935576 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 04:42:09.935780 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:42:09.936143 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260924 04:42:09.938565 10971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260924 04:42:09.938853 10971 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:09.939426 10971 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260924 04:42:09.940541 10644 ts_manager.cc:194] Registered new tserver with Master: fefce902cdcc4e2d92cf1666038fed57 (127.0.98.2:38673)
I20260924 04:42:09.941251 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:42:09.950590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:42:10.065073 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:10.065371 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:10.065418 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:10.069830 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:10.069905 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:10.069967 10976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:42:10.075714 10976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:10.076910 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:10.077927 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:10.087869 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:10.087869 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:10.087869 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:10.088060 10976 server_base.cc:1034] running on GCE node
I20260924 04:42:10.088620 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:10.089344 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:10.090512 10976 hybrid_clock.cc:648] HybridClock initialized: now 1790224930090494 us; error 30 us; skew 500 ppm
Sep 24 04:42:10 dist-test-slave-npg2 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:10.093515 10976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:42:10.094574 10976 webserver.cc:460] Webserver started at http://127.0.98.3:41659/ using document root <none> and password file <none>
I20260924 04:42:10.095197 10976 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:10.095268 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:10.095497 10976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:42:10.097635 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "42a12599fda940ecac56efe064ba4b9a"
format_stamp: "Formatted at 2026-09-24 04:42:10 on dist-test-slave-npg2"
server_key: "f12c15b3d0ff827bfe675c5d46a6dc46"
server_key_iv: "c0b19c7ec13ab3884d1b73f763633e78"
server_key_version: "encryptionkey@0"
I20260924 04:42:10.098208 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42a12599fda940ecac56efe064ba4b9a"
format_stamp: "Formatted at 2026-09-24 04:42:10 on dist-test-slave-npg2"
server_key: "f12c15b3d0ff827bfe675c5d46a6dc46"
server_key_iv: "c0b19c7ec13ab3884d1b73f763633e78"
server_key_version: "encryptionkey@0"
I20260924 04:42:10.102087 10976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:42:10.104775 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:10.106081 10976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 04:42:10.106225 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "42a12599fda940ecac56efe064ba4b9a"
format_stamp: "Formatted at 2026-09-24 04:42:10 on dist-test-slave-npg2"
server_key: "f12c15b3d0ff827bfe675c5d46a6dc46"
server_key_iv: "c0b19c7ec13ab3884d1b73f763633e78"
server_key_version: "encryptionkey@0"
I20260924 04:42:10.106402 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:10.126827 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:10.130411 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:10.130584 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:10.131170 10976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:10.132046 10976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:10.132100 10976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:10.132278 10976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:10.132299 10976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:10.142055 10976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45155
I20260924 04:42:10.142078 11104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45155 every 8 connection(s)
I20260924 04:42:10.143051 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:42:10.147563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 10976
I20260924 04:42:10.147713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:42:10.148052 392 external_mini_cluster.cc:1428] Setting key db063f99fad5a851d44d76776c8cf66c
Sep 24 04:42:10 dist-test-slave-npg2 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:10.157483 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.145041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33447 (local address 127.0.98.62:42511)
0924 04:42:10.145208 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.145213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.146156 (+ 943us) server_negotiation.cc:407] Connection header received
0924 04:42:10.147640 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.147644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.147709 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.147838 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:10.149472 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.150015 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.151276 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.151501 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.151611 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.154005 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:10.154015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:10.154018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:10.154040 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:10.155431 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:10.155934 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:10.155939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:10.155941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:10.156021 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:10.156368 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:10.156371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:10.156373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:10.156592 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:10.156701 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.157186 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.157317 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:42:10.158282 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.145527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:33447)
0924 04:42:10.145999 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.146042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:10.147363 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:10.148156 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:10.148165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:10.148661 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:10.149353 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.149372 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.150168 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.150172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.151103 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.151112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.151647 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.151651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.151851 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.152400 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:10.152415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:10.153843 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:10.155546 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:10.155552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:10.155567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:10.155830 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:10.156119 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:10.156122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:10.156124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:10.156264 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:10.156700 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:10.156707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:10.156956 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:42:10.157148 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 04:42:10.157468 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":150,"threads_started":1}
I20260924 04:42:10.159482 11105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260924 04:42:10.159739 11105 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:10.160228 11105 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260924 04:42:10.161302 10644 ts_manager.cc:194] Registered new tserver with Master: 42a12599fda940ecac56efe064ba4b9a (127.0.98.3:45155)
I20260924 04:42:10.161908 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:42:10.162032 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33447
I20260924 04:42:10.171607 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.164094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.0.98.62:42511)
0924 04:42:10.164220 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.164224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.164416 (+ 192us) server_negotiation.cc:407] Connection header received
0924 04:42:10.164496 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.164498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.164531 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.164598 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:10.165345 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.165868 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.166719 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.166940 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.167025 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.168461 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:10.168463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:10.168465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:10.168487 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:10.169826 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:10.170311 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:10.170314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:10.170316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:10.170389 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:10.170644 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:10.170646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:10.170647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:10.170916 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:10.171019 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.171392 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.171487 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:42:10.174507 10644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:42:10.177589 10644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:42:10.192953 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.186770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45241 (local address 127.0.0.1:59104)
0924 04:42:10.187875 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.187885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:10.187987 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:10.188616 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:10.188621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:10.188639 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:10.188727 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.188732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.189918 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.189922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.191336 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.191345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.192380 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.192383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.192497 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.192547 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:42:10.192654 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:42:10.192792 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":135,"thread_start_us":96,"threads_started":1}
I20260924 04:42:10.193070 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.186758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.98.3:45155)
0924 04:42:10.187742 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.187755 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.187779 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:42:10.187860 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.187867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.188088 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.188242 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:10.188554 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.189808 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.191822 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.192503 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.192621 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.192768 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.192827 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.192916 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":93,"threads_started":1}
I20260924 04:42:10.193557 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.186391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45155 (local address 127.0.0.1:46080)
0924 04:42:10.187466 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.187520 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:10.187661 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:10.188244 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:10.188250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:10.188271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:10.188418 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.188425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.189963 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.189967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.191670 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.191680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.192616 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.192620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.192714 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.192736 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:42:10.192802 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:42:10.192873 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":299,"thread_start_us":90,"threads_started":1}
I20260924 04:42:10.193657 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.187250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.0.98.1:45241)
0924 04:42:10.187631 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.187646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.188062 (+ 416us) server_negotiation.cc:407] Connection header received
0924 04:42:10.188144 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.188152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.188329 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.188467 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:10.188851 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.189615 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.191456 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.192166 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.192357 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.192514 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.193389 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.193519 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":87,"threads_started":1}
I20260924 04:42:10.194264 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.187115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.98.2:38673)
0924 04:42:10.187556 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.187568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.187964 (+ 396us) server_negotiation.cc:407] Connection header received
0924 04:42:10.188046 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.188053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.188248 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.188397 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:10.188679 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.189807 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.191804 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.192857 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.192950 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.193989 (+ 1039us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.194044 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.194135 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260924 04:42:10.194520 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.186560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38673 (local address 127.0.0.1:60674)
0924 04:42:10.187842 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.187855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:10.187949 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:10.188430 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:10.188433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:10.188449 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:10.188556 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.188562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.189963 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.189967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.191670 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.191680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.193475 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.193479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.193565 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.193596 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:42:10.193672 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:42:10.193736 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":110,"thread_start_us":91,"threads_started":1}
I20260924 04:42:10.196799 10771 tablet_service.cc:1467] Processing CreateTablet for tablet 9aca73bfbb5c4a7da0ce26da10869ad1 (DEFAULT_TABLE table=test-table [id=9f047719447a476ab4ba594e8acf26c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:42:10.196795 11039 tablet_service.cc:1467] Processing CreateTablet for tablet 9aca73bfbb5c4a7da0ce26da10869ad1 (DEFAULT_TABLE table=test-table [id=9f047719447a476ab4ba594e8acf26c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:42:10.196770 10905 tablet_service.cc:1467] Processing CreateTablet for tablet 9aca73bfbb5c4a7da0ce26da10869ad1 (DEFAULT_TABLE table=test-table [id=9f047719447a476ab4ba594e8acf26c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:42:10.197870 10771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aca73bfbb5c4a7da0ce26da10869ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:10.197875 11039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aca73bfbb5c4a7da0ce26da10869ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:10.197873 10905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aca73bfbb5c4a7da0ce26da10869ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:10.204133 11122 tablet_bootstrap.cc:492] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Bootstrap starting.
I20260924 04:42:10.205746 11123 tablet_bootstrap.cc:492] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: Bootstrap starting.
I20260924 04:42:10.207058 11122 tablet_bootstrap.cc:654] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:10.207499 11124 tablet_bootstrap.cc:492] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Bootstrap starting.
I20260924 04:42:10.207994 11122 log.cc:824] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:10.208571 11123 tablet_bootstrap.cc:654] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:10.209436 11123 log.cc:824] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:10.210391 11122 tablet_bootstrap.cc:492] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: No bootstrap required, opened a new log
I20260924 04:42:10.210538 11122 ts_tablet_manager.cc:1397] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:42:10.210634 11124 tablet_bootstrap.cc:654] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:10.211658 11123 tablet_bootstrap.cc:492] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: No bootstrap required, opened a new log
I20260924 04:42:10.211858 11123 ts_tablet_manager.cc:1397] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:42:10.211913 11124 log.cc:824] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:10.213933 11124 tablet_bootstrap.cc:492] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: No bootstrap required, opened a new log
I20260924 04:42:10.213778 11122 raft_consensus.cc:348] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.214037 11122 raft_consensus.cc:374] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:10.214094 11122 raft_consensus.cc:729] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fefce902cdcc4e2d92cf1666038fed57, State: Initialized, Role: FOLLOWER
I20260924 04:42:10.214130 11124 ts_tablet_manager.cc:1397] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:42:10.214601 11122 consensus_queue.cc:260] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.215669 11122 ts_tablet_manager.cc:1428] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:42:10.216029 10971 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260924 04:42:10.216226 11123 raft_consensus.cc:348] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.216492 11123 raft_consensus.cc:374] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:10.216549 11123 raft_consensus.cc:729] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caf471898244123be9d6a5f97efad6f, State: Initialized, Role: FOLLOWER
I20260924 04:42:10.217031 11123 consensus_queue.cc:260] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.217751 10837 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260924 04:42:10.218027 11123 ts_tablet_manager.cc:1428] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:42:10.218993 11124 raft_consensus.cc:348] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.219337 11124 raft_consensus.cc:374] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:10.219400 11124 raft_consensus.cc:729] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a12599fda940ecac56efe064ba4b9a, State: Initialized, Role: FOLLOWER
I20260924 04:42:10.219893 11124 consensus_queue.cc:260] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.220935 11124 ts_tablet_manager.cc:1428] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:42:10.221285 11105 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
W20260924 04:42:10.394230 11106 tablet.cc:2369] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:42:10.413236 10972 tablet.cc:2369] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:42:10.426841 10838 tablet.cc:2369] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:42:10.606158 11129 raft_consensus.cc:482] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:42:10.606452 11129 raft_consensus.cc:504] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.607935 11129 leader_election.cc:288] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42a12599fda940ecac56efe064ba4b9a (127.0.98.3:45155), fefce902cdcc4e2d92cf1666038fed57 (127.0.98.2:38673)
I20260924 04:42:10.612262 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.608382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60689 (local address 127.0.98.3:45155)
0924 04:42:10.608533 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.608539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.608765 (+ 226us) server_negotiation.cc:407] Connection header received
0924 04:42:10.608952 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.608956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.609008 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.609093 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:10.609374 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.609991 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.611297 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.611707 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.611825 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.611898 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.612063 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.612129 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:42:10.612296 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.608443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52941 (local address 127.0.98.2:38673)
0924 04:42:10.608638 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.608645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.608878 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:42:10.609034 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.609039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.609086 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.609172 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:10.609470 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.610062 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.611312 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.611751 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.611867 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.611931 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.612110 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.612174 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 04:42:10.612322 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.608266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45155 (local address 127.0.98.1:60689)
0924 04:42:10.608652 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.608668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:10.608843 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:10.609104 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:10.609107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:10.609121 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:10.609269 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.609274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.610092 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.610099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.611167 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.611176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.611831 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.611834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.611931 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.611962 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:42:10.612047 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:42:10.612118 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":48,"thread_start_us":123,"threads_started":1}
I20260924 04:42:10.612334 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.608303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38673 (local address 127.0.98.1:52941)
0924 04:42:10.608777 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.608788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:10.608914 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:10.609179 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:10.609182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:10.609204 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:10.609356 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.609361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.610165 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.610170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.611183 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:10.611193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.611870 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.611873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:10.611980 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.612001 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:42:10.612098 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:42:10.612164 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":41,"thread_start_us":95,"threads_started":1}
I20260924 04:42:10.612972 10925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aca73bfbb5c4a7da0ce26da10869ad1" candidate_uuid: "6caf471898244123be9d6a5f97efad6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fefce902cdcc4e2d92cf1666038fed57" is_pre_election: true
I20260924 04:42:10.613322 10925 raft_consensus.cc:2454] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caf471898244123be9d6a5f97efad6f in term 0.
I20260924 04:42:10.613322 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aca73bfbb5c4a7da0ce26da10869ad1" candidate_uuid: "6caf471898244123be9d6a5f97efad6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a12599fda940ecac56efe064ba4b9a" is_pre_election: true
I20260924 04:42:10.613595 11059 raft_consensus.cc:2454] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caf471898244123be9d6a5f97efad6f in term 0.
I20260924 04:42:10.613824 10725 leader_election.cc:302] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caf471898244123be9d6a5f97efad6f, fefce902cdcc4e2d92cf1666038fed57; no voters:
I20260924 04:42:10.614106 11129 raft_consensus.cc:2781] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:42:10.614220 11129 raft_consensus.cc:482] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:42:10.614269 11129 raft_consensus.cc:3037] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:10.615360 11129 raft_consensus.cc:504] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.615720 11129 leader_election.cc:288] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [CANDIDATE]: Term 1 election: Requested vote from peers 42a12599fda940ecac56efe064ba4b9a (127.0.98.3:45155), fefce902cdcc4e2d92cf1666038fed57 (127.0.98.2:38673)
I20260924 04:42:10.616083 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aca73bfbb5c4a7da0ce26da10869ad1" candidate_uuid: "6caf471898244123be9d6a5f97efad6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a12599fda940ecac56efe064ba4b9a"
I20260924 04:42:10.616101 10925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aca73bfbb5c4a7da0ce26da10869ad1" candidate_uuid: "6caf471898244123be9d6a5f97efad6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fefce902cdcc4e2d92cf1666038fed57"
I20260924 04:42:10.616225 11059 raft_consensus.cc:3037] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:10.616245 10925 raft_consensus.cc:3037] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:10.617297 10925 raft_consensus.cc:2454] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6caf471898244123be9d6a5f97efad6f in term 1.
I20260924 04:42:10.617341 11059 raft_consensus.cc:2454] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6caf471898244123be9d6a5f97efad6f in term 1.
I20260924 04:42:10.617574 10725 leader_election.cc:302] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caf471898244123be9d6a5f97efad6f, fefce902cdcc4e2d92cf1666038fed57; no voters:
I20260924 04:42:10.617796 11129 raft_consensus.cc:2781] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:42:10.618026 11129 raft_consensus.cc:686] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 1 LEADER]: Becoming Leader. State: Replica: 6caf471898244123be9d6a5f97efad6f, State: Running, Role: LEADER
I20260924 04:42:10.618402 11129 consensus_queue.cc:237] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } }
I20260924 04:42:10.621771 10646 catalog_manager.cc:5495] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6caf471898244123be9d6a5f97efad6f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6caf471898244123be9d6a5f97efad6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6caf471898244123be9d6a5f97efad6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45241 } health_report { overall_health: HEALTHY } } }
I20260924 04:42:10.651185 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.647011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.98.1:45241)
0924 04:42:10.647181 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:10.647185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:10.647201 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:42:10.647301 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:10.647304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:10.647367 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:10.647489 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:42:10.647723 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.648249 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.649266 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:10.649565 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:10.649695 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:10.650063 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:42:10.650735 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:42:10.650869 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:10.650940 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:42:10.651004 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:42:10.657429 11059 raft_consensus.cc:1261] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 1 FOLLOWER]: Refusing update from remote peer 6caf471898244123be9d6a5f97efad6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:42:10.657429 10925 raft_consensus.cc:1261] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 1 FOLLOWER]: Refusing update from remote peer 6caf471898244123be9d6a5f97efad6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:42:10.658147 11129 consensus_queue.cc:1035] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a12599fda940ecac56efe064ba4b9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:42:10.658396 11133 consensus_queue.cc:1035] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fefce902cdcc4e2d92cf1666038fed57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:42:10.667438 11137 mvcc.cc:205] Tried to move back new op lower bound from 7332761315965485056 to 7332761315815849984. Current Snapshot: MvccSnapshot[applied={T|T < 7332761315965485056}]
I20260924 04:42:10.673691 10646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:42:10.673846 10646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:42:10.676561 10646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dbf8da0ea4a4f92a43d507ae1c21599: Sending DeleteTablet for 3 replicas of tablet 9aca73bfbb5c4a7da0ce26da10869ad1
I20260924 04:42:10.677264 11039 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aca73bfbb5c4a7da0ce26da10869ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:42:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46080
I20260924 04:42:10.679665 10771 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aca73bfbb5c4a7da0ce26da10869ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:42:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59104
I20260924 04:42:10.679665 10905 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aca73bfbb5c4a7da0ce26da10869ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:42:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60674
I20260924 04:42:10.680181 11147 tablet_replica.cc:330] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f: stopping tablet replica
I20260924 04:42:10.680217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10708
I20260924 04:42:10.680569 11147 raft_consensus.cc:2229] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 6caf471898244123be9d6a5f97efad6f [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:42:10.680941 11146 tablet_replica.cc:330] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: stopping tablet replica
I20260924 04:42:10.681207 11146 raft_consensus.cc:2229] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:42:10.681515 11146 raft_consensus.cc:2258] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:42:10.681890 11148 tablet_replica.cc:330] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: stopping tablet replica
I20260924 04:42:10.682193 11148 raft_consensus.cc:2229] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:42:10.682525 11148 raft_consensus.cc:2258] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:42:10.683405 11146 ts_tablet_manager.cc:1905] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:42:10.684024 11148 ts_tablet_manager.cc:1905] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:42:10.686401 11146 ts_tablet_manager.cc:1918] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:42:10.686506 11146 log.cc:1196] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/9aca73bfbb5c4a7da0ce26da10869ad1
I20260924 04:42:10.686834 11146 ts_tablet_manager.cc:1939] T 9aca73bfbb5c4a7da0ce26da10869ad1 P 42a12599fda940ecac56efe064ba4b9a: Deleting consensus metadata
I20260924 04:42:10.686843 11148 ts_tablet_manager.cc:1918] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:42:10.686962 11148 log.cc:1196] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/9aca73bfbb5c4a7da0ce26da10869ad1
I20260924 04:42:10.687287 11148 ts_tablet_manager.cc:1939] T 9aca73bfbb5c4a7da0ce26da10869ad1 P fefce902cdcc4e2d92cf1666038fed57: Deleting consensus metadata
I20260924 04:42:10.688364 10633 catalog_manager.cc:4841] TS 42a12599fda940ecac56efe064ba4b9a (127.0.98.3:45155): tablet 9aca73bfbb5c4a7da0ce26da10869ad1 (table test-table [id=9f047719447a476ab4ba594e8acf26c8]) successfully deleted
I20260924 04:42:10.688505 10631 catalog_manager.cc:4841] TS fefce902cdcc4e2d92cf1666038fed57 (127.0.98.2:38673): tablet 9aca73bfbb5c4a7da0ce26da10869ad1 (table test-table [id=9f047719447a476ab4ba594e8acf26c8]) successfully deleted
W20260924 04:42:10.689119 10630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45241 (error 108)
I20260924 04:42:10.689390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10842
I20260924 04:42:10.690368 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:10.690097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45241 (local address 127.0.0.1:59130)
0924 04:42:10.690209 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:10.690293 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:42:10.690516 10630 catalog_manager.cc:4584] TS 6caf471898244123be9d6a5f97efad6f (127.0.98.1:45241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aca73bfbb5c4a7da0ce26da10869ad1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45241: connect: Connection refused (error 111)
I20260924 04:42:10.696235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10976
I20260924 04:42:10.701719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 10614
2026-09-24T04:42:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2700 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:42:10 dist-test-slave-npg2 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:42:10 dist-test-slave-npg2 krb5kdc[11152](info): set up 2 sockets
Sep 24 04:42:10 dist-test-slave-npg2 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260924 04:42:11.764166 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:42:11 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:42:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:42:11.940074 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhiQnGj/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:42:14.680 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:42:14.680 UTC [11182] LOG: listening on IPv4 address "127.0.98.20", port 56825
2026-09-24 04:42:14.684 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.56825"
2026-09-24 04:42:14.720 UTC [11185] LOG: database system was shut down at 2026-09-24 04:42:13 UTC
2026-09-24 04:42:14.724 UTC [11182] LOG: database system is ready to accept connections
I20260924 04:42:15.707995 392 mini_postgres.cc:96] Postgres bound to 56825
2026-09-24 04:42:15.714 UTC [11193] FATAL: database "slave" does not exist
127.0.98.20:56825 - accepting connections
I20260924 04:42:15.715232 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:42:15.731261 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:42:16.281286 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:42:16.281395 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-admin
2026-09-24 04:42:16,572 [I] DB FLAVOR :POSTGRES
2026-09-24 04:42:16,573 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:42:16,573 [I] Checking connection..
2026-09-24 04:42:16,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:42:16,850 [I] Checking connection passed.
2026-09-24 04:42:16,851 [I] --------- Verifying version history table ---------
2026-09-24 04:42:16,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:42:17,097 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:42:17,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:42:17,333 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:42:17,334 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:42:17,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:42:17.578 UTC [11257] WARNING: there is no transaction in progress
2026-09-24 04:42:17,584 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:42:17,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:42:17,844 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:42:17,844 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:42:17,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:42:18,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:42:18,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-npg2.c.gcp-upstream.internal','N') ;"
2026-09-24 04:42:18,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:42:18,867 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:42:18,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:42:19,124 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:42:19,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:42:19,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:42:19,629 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:42:19,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:42:20.683 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.702 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.720 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.738 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.755 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.774 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.835 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.842 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.859 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.870 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.882 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.897 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.907 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.914 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.922 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:20.930 UTC [11365] WARNING: there is no transaction in progress
2026-09-24 04:42:21,658 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:42:21,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:42:21,888 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:42:21,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:42:22,134 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:42:22,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:42:22,376 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:42:22,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:42:22,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:42:22,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-npg2.c.gcp-upstream.internal';"
2026-09-24 04:42:23,109 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:42:23,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:42:23,353 [I] DB_PATCHES have already been applied
I20260924 04:42:23.359061 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhiQnGj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:42:23.359195 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:42:23.362785 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52417
Sep 24, 2026 4:42:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:42:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:42:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:42:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:42:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:42:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:42:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:42:24.493062 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.778s user 0.000s sys 0.011s
Sep 24, 2026 4:42:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:42:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:42:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:42:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:42:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:42:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:42:46.459296 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:42:46 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224966, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:42:46.553470 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:42:46.563875 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:42:47.559478 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:42:47.559580 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms
I20260924 04:42:47.559702 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56825
2026-09-24 04:42:47,741 [I] DB FLAVOR :POSTGRES
2026-09-24 04:42:47,741 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:42:47,741 [I] Checking connection
2026-09-24 04:42:47,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:42:47,993 [I] connection success
2026-09-24 04:42:47,994 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:42:47,994 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:42:47,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:42:48,238 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:42:48,239 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:42:48,239 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:42:48,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56825/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:42:48,527 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:42:49.050076 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:42:49.233251 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:42:49.279047 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:42:49 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224966, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:42:49.730319 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:42:49.730494 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:42:49.730526 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:42:49.735009 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260924 04:42:49.735179 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53041
Sep 24, 2026 4:42:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:42:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:42:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:42:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:42:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:42:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:42:52.055042 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.502s user 0.012s sys 0.154s
I20260924 04:42:52.055210 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:42:52.055357 392 mini_ranger_kms.cc:401] 127.0.98.20:53041/kms/v1/keys
Sep 24 04:42:52 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224972, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:42:52 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224972, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:42:53 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224972, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:42:55 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:42:55.478530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46309
--ranger_config_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53041/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:42:55.610714 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:55.610993 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:55.611071 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:55.615111 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:42:55.615188 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:55.615208 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:55.615224 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:42:55.615240 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:42:55.620707 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53041/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46309
--ranger_config_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:55.621691 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:55.622687 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:55.631937 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:55.632063 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:55.632166 11596 server_base.cc:1034] running on GCE node
W20260924 04:42:55.632305 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:55.632889 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:55.634054 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:55.635205 11596 hybrid_clock.cc:648] HybridClock initialized: now 1790224975635169 us; error 47 us; skew 500 ppm
Sep 24 04:42:55 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:55.643888 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:42:55.644976 11596 webserver.cc:460] Webserver started at http://127.0.98.62:42401/ using document root <none> and password file <none>
I20260924 04:42:55.645624 11596 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:55.645690 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:55.645911 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:42:55 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:42:55.713809 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "154ef06158024f4ca4eef0c755c9defa"
format_stamp: "Formatted at 2026-09-24 04:42:55 on dist-test-slave-npg2"
server_key: "1cdd2306e46e416dd6c5d71abaef744b"
server_key_iv: "f053fe92953c3918aaa89eec7e6c456b"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:55.714444 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "154ef06158024f4ca4eef0c755c9defa"
format_stamp: "Formatted at 2026-09-24 04:42:55 on dist-test-slave-npg2"
server_key: "1cdd2306e46e416dd6c5d71abaef744b"
server_key_iv: "f053fe92953c3918aaa89eec7e6c456b"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:55.718662 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 04:42:55.737717 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:55.739230 11596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:42:55.739396 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "154ef06158024f4ca4eef0c755c9defa"
format_stamp: "Formatted at 2026-09-24 04:42:55 on dist-test-slave-npg2"
server_key: "1cdd2306e46e416dd6c5d71abaef744b"
server_key_iv: "f053fe92953c3918aaa89eec7e6c456b"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:55.739530 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:55.752470 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:55.755666 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:55.755877 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:55.764740 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46309
I20260924 04:42:55.764756 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46309 every 8 connection(s)
I20260924 04:42:55.765758 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:42:55.767302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 11596
I20260924 04:42:55.767441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:42:55.767788 392 external_mini_cluster.cc:1428] Setting key 36f7092cce446b47fceffd3090c55e61
I20260924 04:42:55.769062 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:55.775668 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa: Bootstrap starting.
I20260924 04:42:55.778172 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa: Neither blocks nor log segments found. Creating new log.
Sep 24 04:42:55 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:55.779156 11664 log.cc:824] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:55.781318 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa: No bootstrap required, opened a new log
I20260924 04:42:55.786135 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154ef06158024f4ca4eef0c755c9defa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46309 } }
I20260924 04:42:55.786427 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:55.786491 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154ef06158024f4ca4eef0c755c9defa, State: Initialized, Role: FOLLOWER
I20260924 04:42:55.787065 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154ef06158024f4ca4eef0c755c9defa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46309 } }
I20260924 04:42:55.787408 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:42:55.787492 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:42:55.787601 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:55.788460 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:55.769333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.62:46309)
0924 04:42:55.769814 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:55.769833 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:55.769875 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:42:55.770734 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:55.770756 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:55.771214 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:55.771569 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:55.772562 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:55.774041 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:55.775375 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:55.776101 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:55.776256 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:55.779595 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:55.779602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:55.779618 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:55.779651 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:55.784901 (+ 5250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:55.785549 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:55.785556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:55.785563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:55.785706 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:55.786025 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:55.786028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:55.786030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:55.786574 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:55.786776 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:55.787098 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 04:42:55.787427 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":163,"threads_started":1}
I20260924 04:42:55.789623 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154ef06158024f4ca4eef0c755c9defa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46309 } }
I20260924 04:42:55.789994 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 154ef06158024f4ca4eef0c755c9defa; no voters:
I20260924 04:42:55.790349 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:42:55.790541 11669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:42:55.790786 11669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [term 1 LEADER]: Becoming Leader. State: Replica: 154ef06158024f4ca4eef0c755c9defa, State: Running, Role: LEADER
I20260924 04:42:55.791198 11669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154ef06158024f4ca4eef0c755c9defa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46309 } }
I20260924 04:42:55.791963 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:42:55.793265 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 154ef06158024f4ca4eef0c755c9defa. Latest consensus state: current_term: 1 leader_uuid: "154ef06158024f4ca4eef0c755c9defa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154ef06158024f4ca4eef0c755c9defa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46309 } } }
I20260924 04:42:55.793396 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:55.793316 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "154ef06158024f4ca4eef0c755c9defa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154ef06158024f4ca4eef0c755c9defa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46309 } } }
I20260924 04:42:55.793567 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa [sys.catalog]: This master's current role is: LEADER
I20260924 04:42:55.794381 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:42:55.799015 11678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:42:55.803067 11678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:42:55.812999 11678 catalog_manager.cc:1347] Generated new cluster ID: b7655dfbfe30449eb4d57dcd74272ebb
I20260924 04:42:55.813081 11678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:42:55.842518 11678 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:42:55.845216 11678 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:42:55.855391 11678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa: Generated new TSK 0
I20260924 04:42:55.857405 11678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:42:56.602380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46309
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:42:56.730490 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:56.730871 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:56.730939 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:56.737207 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:56.737367 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:56.737507 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:42:56.745217 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:56.746693 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:56.748052 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:56.758160 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:56.758160 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:56.758301 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:56.758651 11709 server_base.cc:1034] running on GCE node
I20260924 04:42:56.759125 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:56.759862 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:56.761227 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790224976761058 us; error 47 us; skew 500 ppm
Sep 24 04:42:56 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:56.764698 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:42:56.765877 11709 webserver.cc:460] Webserver started at http://127.0.98.1:37159/ using document root <none> and password file <none>
I20260924 04:42:56.766629 11709 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:56.766790 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:56.767145 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:42:56 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:42:56.790234 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "4fcdb08e27c84af7870f83feef56e81a"
format_stamp: "Formatted at 2026-09-24 04:42:56 on dist-test-slave-npg2"
server_key: "9a852006126f0bcf62e73950ba4deffe"
server_key_iv: "cdcf1070b368ba4050d3f1234ebe3875"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:56.791071 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4fcdb08e27c84af7870f83feef56e81a"
format_stamp: "Formatted at 2026-09-24 04:42:56 on dist-test-slave-npg2"
server_key: "9a852006126f0bcf62e73950ba4deffe"
server_key_iv: "cdcf1070b368ba4050d3f1234ebe3875"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:56.796684 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:42:56.835493 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:56.838763 11709 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.000s
I20260924 04:42:56.839007 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "4fcdb08e27c84af7870f83feef56e81a"
format_stamp: "Formatted at 2026-09-24 04:42:56 on dist-test-slave-npg2"
server_key: "9a852006126f0bcf62e73950ba4deffe"
server_key_iv: "cdcf1070b368ba4050d3f1234ebe3875"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:56.839316 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:56.865157 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:56.869437 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:56.869731 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:56.870601 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:56.871878 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:56.872402 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260924 04:42:56.872543 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:56.872632 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:56.886713 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32863
I20260924 04:42:56.886776 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32863 every 8 connection(s)
I20260924 04:42:56.888109 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:42:56.896884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 11709
I20260924 04:42:56.897032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:42:56.897328 392 external_mini_cluster.cc:1428] Setting key b0af0a2c384521e548cd137a9067c5d4
Sep 24 04:42:56 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:56.909140 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:56.890738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46309 (local address 127.0.98.1:38695)
0924 04:42:56.891698 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:56.891756 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:56.892831 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:56.895793 (+ 2962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:56.895803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:56.896182 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:56.896900 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:56.896921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:56.898170 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:56.898173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:56.899669 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:56.899680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:56.900485 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:56.900492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:56.900698 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:56.901413 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:56.901434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:56.903305 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:56.905680 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:56.905686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:56.905705 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:56.906062 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:56.906380 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:56.906384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:56.906386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:56.906581 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:56.907061 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:56.907068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:56.907456 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 04:42:56.907659 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 04:42:56.908012 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":187,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:42:56.909916 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:56.891271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38695 (local address 127.0.98.62:46309)
0924 04:42:56.895340 (+ 4069us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:56.895345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:56.895360 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:42:56.895405 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:56.895409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:56.895459 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:56.895563 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:56.897091 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:56.898008 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:56.899813 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:56.900181 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:56.900288 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:56.903563 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:56.903568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:56.903570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:56.903599 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:56.905541 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:56.906183 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:56.906187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:56.906189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:56.906281 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:56.906691 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:56.906694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:56.906696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:56.906922 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:56.907011 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:56.909617 (+ 2606us) server_negotiation.cc:299] Negotiation successful
0924 04:42:56.909751 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3982,"thread_start_us":85,"threads_started":1}
I20260924 04:42:56.911635 11842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46309
I20260924 04:42:56.911911 11842 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:56.912528 11842 heartbeater.cc:511] Master 127.0.98.62:46309 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:42:56.914439 11628 ts_manager.cc:194] Registered new tserver with Master: 4fcdb08e27c84af7870f83feef56e81a (127.0.98.1:32863)
I20260924 04:42:56.916075 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:42:56.935968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46309
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:42:56 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:42:57.102699 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:57.103164 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:57.103240 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:57.109426 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:57.109510 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:57.109611 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:42:57.117908 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:57.119585 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:57.120967 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:57.134349 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:57.134392 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:57.134907 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:57.135507 11848 server_base.cc:1034] running on GCE node
I20260924 04:42:57.135975 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:57.136754 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:57.137962 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790224977137939 us; error 37 us; skew 500 ppm
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:57.142097 11848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:42:57.143289 11848 webserver.cc:460] Webserver started at http://127.0.98.2:44607/ using document root <none> and password file <none>
I20260924 04:42:57.144040 11848 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:57.144142 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:57.144435 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:42:57.171528 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "23a02386126d461599937b573de8f842"
format_stamp: "Formatted at 2026-09-24 04:42:57 on dist-test-slave-npg2"
server_key: "1025ca455f4da0ce1b1b5d0f32a96668"
server_key_iv: "0803e9cefe31f3a6ce672954e8b9270d"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:57.172731 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23a02386126d461599937b573de8f842"
format_stamp: "Formatted at 2026-09-24 04:42:57 on dist-test-slave-npg2"
server_key: "1025ca455f4da0ce1b1b5d0f32a96668"
server_key_iv: "0803e9cefe31f3a6ce672954e8b9270d"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:57.178949 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 04:42:57.209033 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:57.210949 11848 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.005s sys 0.000s
I20260924 04:42:57.211150 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "23a02386126d461599937b573de8f842"
format_stamp: "Formatted at 2026-09-24 04:42:57 on dist-test-slave-npg2"
server_key: "1025ca455f4da0ce1b1b5d0f32a96668"
server_key_iv: "0803e9cefe31f3a6ce672954e8b9270d"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:57.211329 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:57.233395 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:57.237339 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:57.237599 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:57.238374 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:57.239518 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:57.239622 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:57.239737 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:57.239799 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:57.253623 11848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41177
I20260924 04:42:57.254938 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:42:57.257858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 11848
I20260924 04:42:57.257997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:42:57.258343 392 external_mini_cluster.cc:1428] Setting key 3a0fe06f75678ae43131772518834c42
I20260924 04:42:57.256699 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41177 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:57.288918 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.265355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46393 (local address 127.0.98.62:46309)
0924 04:42:57.265489 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.265494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.266028 (+ 534us) server_negotiation.cc:407] Connection header received
0924 04:42:57.270458 (+ 4430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.270463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.270529 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.270591 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:57.272390 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.273244 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.279978 (+ 6734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.280399 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.280494 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.283931 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:57.283935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:57.283938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:57.283973 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:57.286002 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:57.286682 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:57.286686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:57.286688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:57.286774 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:57.287321 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:57.287325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:57.287326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:57.287586 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:57.287687 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.288577 (+ 890us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.288722 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:42:57.293623 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.264134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46309 (local address 127.0.98.2:46393)
0924 04:42:57.265851 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.265915 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.267065 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.271043 (+ 3978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.271065 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.271455 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:57.272203 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.272226 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.273411 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.273414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.274838 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.274849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.280773 (+ 5924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.280776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.281023 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.281780 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:57.281816 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:57.283691 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:57.286167 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:57.286174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:57.286195 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:57.286558 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:57.286878 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:57.286880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:57.286882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:57.287206 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:57.287753 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:57.287760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:57.288155 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.289517 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.292111 (+ 2594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"thread_start_us":179,"threads_started":1}
I20260924 04:42:57.295387 11978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46309
I20260924 04:42:57.295662 11978 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:57.296248 11978 heartbeater.cc:511] Master 127.0.98.62:46309 requested a full tablet report, sending...
I20260924 04:42:57.297449 11628 ts_manager.cc:194] Registered new tserver with Master: 23a02386126d461599937b573de8f842 (127.0.98.2:41177)
I20260924 04:42:57.298197 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:42:57.319407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46309
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:42:57.484274 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:42:57.484761 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:42:57.484890 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:42:57.491397 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:42:57.491566 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:42:57.491699 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:42:57.499807 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:42:57.501543 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:42:57.502945 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:42:57.516281 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:42:57.517236 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:57.518151 11983 server_base.cc:1034] running on GCE node
W20260924 04:42:57.523793 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:42:57.524451 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:42:57.525323 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:42:57.526568 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790224977526503 us; error 87 us; skew 500 ppm
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:42:57.530833 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:42:57.532122 11983 webserver.cc:460] Webserver started at http://127.0.98.3:34757/ using document root <none> and password file <none>
I20260924 04:42:57.532938 11983 fs_manager.cc:359] Metadata directory not provided
I20260924 04:42:57.533083 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:42:57.533578 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:42:57.552428 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "2fc7b93b5c394404b9a40d619e427bdb"
format_stamp: "Formatted at 2026-09-24 04:42:57 on dist-test-slave-npg2"
server_key: "20eb870ba1e34e8a1d20804fa0d263b0"
server_key_iv: "32322443150edcc4a1fb3abf45800ad3"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:57.553191 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2fc7b93b5c394404b9a40d619e427bdb"
format_stamp: "Formatted at 2026-09-24 04:42:57 on dist-test-slave-npg2"
server_key: "20eb870ba1e34e8a1d20804fa0d263b0"
server_key_iv: "32322443150edcc4a1fb3abf45800ad3"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:57.559320 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 04:42:57.583637 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:57.585584 11983 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.001s
I20260924 04:42:57.585731 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "2fc7b93b5c394404b9a40d619e427bdb"
format_stamp: "Formatted at 2026-09-24 04:42:57 on dist-test-slave-npg2"
server_key: "20eb870ba1e34e8a1d20804fa0d263b0"
server_key_iv: "32322443150edcc4a1fb3abf45800ad3"
server_key_version: "kuduclusterkey@0"
I20260924 04:42:57.585875 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:42:57.606674 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:42:57.609704 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:42:57.609932 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:42:57.610572 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:42:57.611537 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:42:57.611604 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:57.611701 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:42:57.611748 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:42:57.622864 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39317
I20260924 04:42:57.623948 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:42:57.627774 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39317 every 8 connection(s)
I20260924 04:42:57.632031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 11983
I20260924 04:42:57.632179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:42:57.632509 392 external_mini_cluster.cc:1428] Setting key 0ac1ad218bc964a0370aaa658af8499a
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:42:57.646944 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.629692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32909 (local address 127.0.98.62:46309)
0924 04:42:57.629813 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.629817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.631732 (+ 1915us) server_negotiation.cc:407] Connection header received
0924 04:42:57.634304 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.634307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.634372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.634422 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:57.636154 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.636925 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.638544 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.638870 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.638937 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.642068 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:57.642072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:57.642075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:57.642102 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:57.643880 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:57.644580 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:57.644599 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:57.644601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:57.644687 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:57.645339 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:57.645342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:57.645344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:57.645556 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:57.645676 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.646636 (+ 960us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.646755 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:42:57.648540 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.630246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46309 (local address 127.0.98.3:32909)
0924 04:42:57.631498 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.631574 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.634043 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.634841 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.634851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.635305 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:42:57.635992 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.636010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.637091 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.637094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.638407 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.638416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.639186 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.639190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.639381 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.640235 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:42:57.640255 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:42:57.641852 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:42:57.644061 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:57.644068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:57.644086 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:57.644453 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:57.644779 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:42:57.644783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:42:57.644785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:42:57.644974 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:42:57.645756 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:42:57.645763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:42:57.646358 (+ 595us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.647164 (+ 806us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.647511 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":165,"threads_started":1}
I20260924 04:42:57.649952 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46309
I20260924 04:42:57.650223 12112 heartbeater.cc:464] Registering TS with master...
I20260924 04:42:57.650815 12112 heartbeater.cc:511] Master 127.0.98.62:46309 requested a full tablet report, sending...
I20260924 04:42:57.652001 11628 ts_manager.cc:194] Registered new tserver with Master: 2fc7b93b5c394404b9a40d619e427bdb (127.0.98.3:39317)
I20260924 04:42:57.652737 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32909
I20260924 04:42:57.658593 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:42:57.670480 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.661029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.0.98.62:46309)
0924 04:42:57.661150 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.661155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.661691 (+ 536us) server_negotiation.cc:407] Connection header received
0924 04:42:57.661914 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.661918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.661977 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.662062 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:42:57.662945 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.663795 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.664999 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.665319 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.665418 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.667165 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:42:57.667169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:42:57.667171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:42:57.667197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:42:57.668774 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:57.669237 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:57.669241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:57.669243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:57.669324 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:42:57.669633 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:42:57.669636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:42:57.669638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:42:57.669840 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:42:57.669971 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.670223 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.670333 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:42:57.673914 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:42:57.678457 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:42:57.698256 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.690506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39317 (local address 127.0.0.1:49272)
0924 04:42:57.691702 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.691715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.691800 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.692862 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.692868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.692893 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:57.693042 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.693049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.694518 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.694521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.696382 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.696393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.697716 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.697719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.697825 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.697863 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.697982 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.698101 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":98,"threads_started":1}
I20260924 04:42:57.700044 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.688451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41177 (local address 127.0.0.1:59682)
0924 04:42:57.690902 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.690954 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.691096 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.693675 (+ 2579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.693678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.693695 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:57.693792 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.693813 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.695904 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.695908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.697088 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.697099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.699615 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.699619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.699714 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.699744 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.699809 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.699878 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"thread_start_us":111,"threads_started":1}
I20260924 04:42:57.701049 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.689292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.0.98.2:41177)
0924 04:42:57.693180 (+ 3888us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.693192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.693217 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:42:57.693290 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.693295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.693498 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.693969 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:57.694108 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.695768 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.697274 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.698088 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.700565 (+ 2477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.700732 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.700792 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.700900 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3784,"thread_start_us":115,"threads_started":1}
I20260924 04:42:57.701601 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.690898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.0.98.3:39317)
0924 04:42:57.692028 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.692039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.692060 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:42:57.692136 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.692144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.692628 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.692875 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:57.693184 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.694384 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.696553 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.697412 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.697513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.697659 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.701348 (+ 3689us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.701460 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":134,"threads_started":1}
I20260924 04:42:57.704468 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.691358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.0.1:59580)
0924 04:42:57.698499 (+ 7141us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.698514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.698588 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.699323 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.699326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.699342 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:57.699432 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.699441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.701766 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.701769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.702863 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.702874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.704104 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.704108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.704189 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.704210 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.704280 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.704339 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7089,"thread_start_us":88,"threads_started":1}
I20260924 04:42:57.704834 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.691531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.0.98.1:32863)
0924 04:42:57.691857 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.691865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.698686 (+ 6821us) server_negotiation.cc:407] Connection header received
0924 04:42:57.698769 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.698775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.699009 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.699180 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:57.699570 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.701191 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.703019 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.703805 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.703909 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.704044 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.704620 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.704709 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260924 04:42:57.704442 11911 tablet_service.cc:1467] Processing CreateTablet for tablet 2321299e1d0645ba9010d37a9b4f867f (DEFAULT_TABLE table=test-table [id=403cfae0140c4e94a24877f308a5b09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:42:57.704443 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 2321299e1d0645ba9010d37a9b4f867f (DEFAULT_TABLE table=test-table [id=403cfae0140c4e94a24877f308a5b09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:42:57.705482 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2321299e1d0645ba9010d37a9b4f867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:57.705502 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2321299e1d0645ba9010d37a9b4f867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:57.710186 11775 tablet_service.cc:1467] Processing CreateTablet for tablet 2321299e1d0645ba9010d37a9b4f867f (DEFAULT_TABLE table=test-table [id=403cfae0140c4e94a24877f308a5b09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:42:57.711273 11775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2321299e1d0645ba9010d37a9b4f867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:42:57.716568 12130 tablet_bootstrap.cc:492] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Bootstrap starting.
I20260924 04:42:57.718478 12131 tablet_bootstrap.cc:492] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Bootstrap starting.
I20260924 04:42:57.721894 12131 tablet_bootstrap.cc:654] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:57.723245 12131 log.cc:824] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:57.724706 12130 tablet_bootstrap.cc:654] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:57.725502 12131 tablet_bootstrap.cc:492] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: No bootstrap required, opened a new log
I20260924 04:42:57.725571 12130 log.cc:824] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:57.725764 12131 ts_tablet_manager.cc:1397] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:42:57.726244 12132 tablet_bootstrap.cc:492] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: Bootstrap starting.
I20260924 04:42:57.728791 12132 tablet_bootstrap.cc:654] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: Neither blocks nor log segments found. Creating new log.
I20260924 04:42:57.729598 12132 log.cc:824] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:42:57.731071 12131 raft_consensus.cc:348] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.731519 12131 raft_consensus.cc:374] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:57.731639 12131 raft_consensus.cc:729] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a02386126d461599937b573de8f842, State: Initialized, Role: FOLLOWER
I20260924 04:42:57.731854 12132 tablet_bootstrap.cc:492] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: No bootstrap required, opened a new log
I20260924 04:42:57.732041 12132 ts_tablet_manager.cc:1397] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 04:42:57.732195 12131 consensus_queue.cc:260] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.733212 12131 ts_tablet_manager.cc:1428] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260924 04:42:57.733583 11978 heartbeater.cc:503] Master 127.0.98.62:46309 was elected leader, sending a full tablet report...
I20260924 04:42:57.735322 12130 tablet_bootstrap.cc:492] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: No bootstrap required, opened a new log
I20260924 04:42:57.735505 12130 ts_tablet_manager.cc:1397] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260924 04:42:57.736336 12132 raft_consensus.cc:348] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.736596 12132 raft_consensus.cc:374] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:57.736651 12132 raft_consensus.cc:729] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fcdb08e27c84af7870f83feef56e81a, State: Initialized, Role: FOLLOWER
I20260924 04:42:57.737291 12132 consensus_queue.cc:260] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.738299 12132 ts_tablet_manager.cc:1428] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:42:57.739262 11842 heartbeater.cc:503] Master 127.0.98.62:46309 was elected leader, sending a full tablet report...
I20260924 04:42:57.739472 12130 raft_consensus.cc:348] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.739784 12130 raft_consensus.cc:374] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:42:57.739879 12130 raft_consensus.cc:729] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc7b93b5c394404b9a40d619e427bdb, State: Initialized, Role: FOLLOWER
I20260924 04:42:57.740357 12130 consensus_queue.cc:260] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.741477 12130 ts_tablet_manager.cc:1428] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:42:57.741848 12112 heartbeater.cc:503] Master 127.0.98.62:46309 was elected leader, sending a full tablet report...
W20260924 04:42:57.759089 11979 tablet.cc:2369] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:42:57 dist-test-slave-npg2 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:42:57.878592 12113 tablet.cc:2369] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:42:57.890377 11843 tablet.cc:2369] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:42:57.983091 12137 raft_consensus.cc:482] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:42:57.983351 12137 raft_consensus.cc:504] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.984726 12137 leader_election.cc:288] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a02386126d461599937b573de8f842 (127.0.98.2:41177), 2fc7b93b5c394404b9a40d619e427bdb (127.0.98.3:39317)
I20260924 04:42:57.990968 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.986345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52119 (local address 127.0.98.2:41177)
0924 04:42:57.986489 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.986493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.986844 (+ 351us) server_negotiation.cc:407] Connection header received
0924 04:42:57.987020 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.987023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.987073 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.987157 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:57.987519 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.988212 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.989708 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.990271 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.990350 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.990419 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.990744 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.990824 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:42:57.991680 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.985924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41177 (local address 127.0.98.1:52119)
0924 04:42:57.986769 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.986781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.986904 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.987289 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.987292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.987307 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:57.987397 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.987402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.988373 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.988375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.989564 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.989575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.990521 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.990524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.990612 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.990634 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.991429 (+ 795us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.991518 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":116,"threads_started":1}
I20260924 04:42:57.992566 11919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2321299e1d0645ba9010d37a9b4f867f" candidate_uuid: "4fcdb08e27c84af7870f83feef56e81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a02386126d461599937b573de8f842" is_pre_election: true
I20260924 04:42:57.992951 11919 raft_consensus.cc:2454] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcdb08e27c84af7870f83feef56e81a in term 0.
I20260924 04:42:57.993245 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.985673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41045 (local address 127.0.98.3:39317)
0924 04:42:57.985858 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:57.985864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:57.986530 (+ 666us) server_negotiation.cc:407] Connection header received
0924 04:42:57.986676 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:57.986680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:57.986737 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:57.986818 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:42:57.987264 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.988336 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.991843 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.992467 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.992574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.992654 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:57.993053 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 04:42:57.993132 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:42:57.993528 11732 leader_election.cc:302] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a02386126d461599937b573de8f842, 4fcdb08e27c84af7870f83feef56e81a; no voters:
I20260924 04:42:57.993680 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:57.985184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39317 (local address 127.0.98.1:41045)
0924 04:42:57.986372 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:57.986388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:42:57.986542 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:42:57.986963 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:42:57.986966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:42:57.986989 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:42:57.987101 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.987108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.988527 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.988530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.989969 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:42:57.989981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:42:57.992742 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:42:57.992745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:42:57.992902 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:57.992924 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:42:57.993423 (+ 499us) client_negotiation.cc:240] Negotiation successful
0924 04:42:57.993496 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":104,"threads_started":1}
I20260924 04:42:57.993883 12137 raft_consensus.cc:2781] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:42:57.993978 12137 raft_consensus.cc:482] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:42:57.994038 12137 raft_consensus.cc:3037] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:57.995512 12065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2321299e1d0645ba9010d37a9b4f867f" candidate_uuid: "4fcdb08e27c84af7870f83feef56e81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b93b5c394404b9a40d619e427bdb" is_pre_election: true
I20260924 04:42:57.995863 12065 raft_consensus.cc:2454] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcdb08e27c84af7870f83feef56e81a in term 0.
I20260924 04:42:57.996518 12137 raft_consensus.cc:504] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:57.996928 12137 leader_election.cc:288] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [CANDIDATE]: Term 1 election: Requested vote from peers 23a02386126d461599937b573de8f842 (127.0.98.2:41177), 2fc7b93b5c394404b9a40d619e427bdb (127.0.98.3:39317)
I20260924 04:42:57.997339 12065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2321299e1d0645ba9010d37a9b4f867f" candidate_uuid: "4fcdb08e27c84af7870f83feef56e81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b93b5c394404b9a40d619e427bdb"
I20260924 04:42:57.997344 11919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2321299e1d0645ba9010d37a9b4f867f" candidate_uuid: "4fcdb08e27c84af7870f83feef56e81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a02386126d461599937b573de8f842"
I20260924 04:42:57.997459 11919 raft_consensus.cc:3037] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:57.997458 12065 raft_consensus.cc:3037] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:42:57.998750 11919 raft_consensus.cc:2454] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcdb08e27c84af7870f83feef56e81a in term 1.
I20260924 04:42:57.998749 12065 raft_consensus.cc:2454] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcdb08e27c84af7870f83feef56e81a in term 1.
I20260924 04:42:57.999078 11729 leader_election.cc:302] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7b93b5c394404b9a40d619e427bdb, 4fcdb08e27c84af7870f83feef56e81a; no voters:
I20260924 04:42:57.999328 12137 raft_consensus.cc:2781] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:42:57.999564 12137 raft_consensus.cc:686] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 1 LEADER]: Becoming Leader. State: Replica: 4fcdb08e27c84af7870f83feef56e81a, State: Running, Role: LEADER
I20260924 04:42:57.999979 12137 consensus_queue.cc:237] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260924 04:42:58.003516 11627 catalog_manager.cc:5495] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fcdb08e27c84af7870f83feef56e81a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4fcdb08e27c84af7870f83feef56e81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fcdb08e27c84af7870f83feef56e81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } health_report { overall_health: HEALTHY } } }
I20260924 04:42:58.075385 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:58.071618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.0.98.1:32863)
0924 04:42:58.071750 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:42:58.071755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:42:58.071769 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:42:58.071908 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:42:58.071910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:42:58.071966 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:42:58.072114 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:42:58.072377 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:58.072895 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:58.073813 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:42:58.074039 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:42:58.074147 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:42:58.074480 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:42:58.074946 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:42:58.075078 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:42:58.075144 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:42:58.075209 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:42:58.081648 11919 raft_consensus.cc:1261] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 1 FOLLOWER]: Refusing update from remote peer 4fcdb08e27c84af7870f83feef56e81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:42:58.081678 12065 raft_consensus.cc:1261] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 1 FOLLOWER]: Refusing update from remote peer 4fcdb08e27c84af7870f83feef56e81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:42:58.082779 12143 consensus_queue.cc:1035] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc7b93b5c394404b9a40d619e427bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:42:58.082994 12137 consensus_queue.cc:1035] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a02386126d461599937b573de8f842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:42:58.091019 12153 mvcc.cc:205] Tried to move back new op lower bound from 7332761510214729728 to 7332761509891026944. Current Snapshot: MvccSnapshot[applied={T|T < 7332761510214729728}]
I20260924 04:42:58.091488 12155 mvcc.cc:205] Tried to move back new op lower bound from 7332761510214729728 to 7332761509891026944. Current Snapshot: MvccSnapshot[applied={T|T < 7332761510214729728}]
I20260924 04:42:58.102996 11628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:42:58.103204 11628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:42:58.105392 11628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 154ef06158024f4ca4eef0c755c9defa: Sending DeleteTablet for 3 replicas of tablet 2321299e1d0645ba9010d37a9b4f867f
I20260924 04:42:58.106213 11911 tablet_service.cc:1514] Processing DeleteTablet for tablet 2321299e1d0645ba9010d37a9b4f867f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:42:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59682
I20260924 04:42:58.106330 11775 tablet_service.cc:1514] Processing DeleteTablet for tablet 2321299e1d0645ba9010d37a9b4f867f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:42:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59580
I20260924 04:42:58.107177 12162 tablet_replica.cc:330] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a: stopping tablet replica
I20260924 04:42:58.107161 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2321299e1d0645ba9010d37a9b4f867f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:42:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49272
I20260924 04:42:58.107628 12162 raft_consensus.cc:2229] T 2321299e1d0645ba9010d37a9b4f867f P 4fcdb08e27c84af7870f83feef56e81a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:42:58.108251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 11709
I20260924 04:42:58.111373 12163 tablet_replica.cc:330] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: stopping tablet replica
I20260924 04:42:58.111699 12163 raft_consensus.cc:2229] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:42:58.112087 12163 raft_consensus.cc:2258] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:42:58.112461 12164 tablet_replica.cc:330] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: stopping tablet replica
I20260924 04:42:58.112730 12164 raft_consensus.cc:2229] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:42:58.112991 12164 raft_consensus.cc:2258] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:42:58.114001 12163 ts_tablet_manager.cc:1905] T 2321299e1d0645ba9010d37a9b4f867f P 23a02386126d461599937b573de8f842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:42:58.114585 12164 ts_tablet_manager.cc:1905] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:42:58.116219 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32863 (error 108)
I20260924 04:42:58.117245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 11848
I20260924 04:42:58.117288 12164 ts_tablet_manager.cc:1918] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:42:58.117368 12164 log.cc:1196] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/2321299e1d0645ba9010d37a9b4f867f
I20260924 04:42:58.117918 12164 ts_tablet_manager.cc:1939] T 2321299e1d0645ba9010d37a9b4f867f P 2fc7b93b5c394404b9a40d619e427bdb: Deleting consensus metadata
I20260924 04:42:58.118146 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:58.117476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.0.1:59594)
0924 04:42:58.117953 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:58.118042 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
W20260924 04:42:58.118546 11614 catalog_manager.cc:4584] TS 4fcdb08e27c84af7870f83feef56e81a (127.0.98.1:32863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2321299e1d0645ba9010d37a9b4f867f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32863: connect: Connection refused (error 111)
I20260924 04:42:58.119185 11612 catalog_manager.cc:4841] TS 2fc7b93b5c394404b9a40d619e427bdb (127.0.98.3:39317): tablet 2321299e1d0645ba9010d37a9b4f867f (table test-table [id=403cfae0140c4e94a24877f308a5b09f]) successfully deleted
I20260924 04:42:58.124485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 11983
I20260924 04:42:58.125872 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:42:58.124643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41177 (local address 127.0.0.1:59686)
0924 04:42:58.125731 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 04:42:58.125788 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1000}
I20260924 04:42:58.130141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 11596
I20260924 04:42:58.137174 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:42:58.482468 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:42:58.482653 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:42:58.827795 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:42:58.828 UTC [11182] LOG: received smart shutdown request
2026-09-24 04:42:58.831 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-09-24 04:42:58.832 UTC [11186] LOG: shutting down
2026-09-24 04:42:59.013 UTC [11182] LOG: database system is shut down
2026-09-24T04:42:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48428 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:42:59 dist-test-slave-npg2 krb5kdc[12174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:42:59 dist-test-slave-npg2 krb5kdc[12174](info): set up 2 sockets
Sep 24 04:42:59 dist-test-slave-npg2 krb5kdc[12174](info): commencing operation
krb5kdc: starting...
W20260924 04:43:00.179438 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:43:00 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:43:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:43:00.357776 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhiQnGj/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:43:03.104 UTC [12204] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:43:03.104 UTC [12204] LOG: listening on IPv4 address "127.0.98.20", port 42781
2026-09-24 04:43:03.108 UTC [12204] LOG: listening on Unix socket "/tmp/.s.PGSQL.42781"
2026-09-24 04:43:03.144 UTC [12207] LOG: database system was shut down at 2026-09-24 04:43:01 UTC
2026-09-24 04:43:03.148 UTC [12204] LOG: database system is ready to accept connections
I20260924 04:43:04.115217 392 mini_postgres.cc:96] Postgres bound to 42781
2026-09-24 04:43:04.121 UTC [12215] FATAL: database "slave" does not exist
127.0.98.20:42781 - accepting connections
I20260924 04:43:04.122538 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:43:04.139016 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:43:04.706264 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:43:04.706336 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-admin
2026-09-24 04:43:04,986 [I] DB FLAVOR :POSTGRES
2026-09-24 04:43:04,986 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:43:04,986 [I] Checking connection..
2026-09-24 04:43:04,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:43:05,258 [I] Checking connection passed.
2026-09-24 04:43:05,258 [I] --------- Verifying version history table ---------
2026-09-24 04:43:05,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:43:05,502 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:43:05,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:43:05,774 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:43:05,775 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:43:05,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:43:06.030 UTC [12279] WARNING: there is no transaction in progress
2026-09-24 04:43:06,037 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:43:06,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:43:06,281 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:43:06,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:43:06,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:43:06,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:43:06,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-npg2.c.gcp-upstream.internal','N') ;"
2026-09-24 04:43:07,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:43:07,298 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:43:07,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:43:07,550 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:43:07,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:43:07,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:43:08,043 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:43:08,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:43:09.108 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.127 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.146 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.164 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.184 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.204 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.269 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.277 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.295 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.306 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.318 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.334 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.346 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.355 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.363 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:09.371 UTC [12387] WARNING: there is no transaction in progress
2026-09-24 04:43:10,163 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:43:10,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:43:10,395 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:43:10,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:43:10,683 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:43:10,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:43:10,992 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:43:10,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:43:11,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:43:11,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-npg2.c.gcp-upstream.internal';"
2026-09-24 04:43:11,758 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:43:11,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:43:12,001 [I] DB_PATCHES have already been applied
I20260924 04:43:12.007294 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhiQnGj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:43:12.007368 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:43:12.010766 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:57547
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:43:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:43:13.171337 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.049s user 0.000s sys 0.011s
Sep 24, 2026 4:43:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:43:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 4:43:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:43:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:43:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:43:35.565119 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:43:35 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225015, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:43:35.673363 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:43:35.684224 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:43:36.655622 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:43:36.655746 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms
I20260924 04:43:36.655769 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42781
2026-09-24 04:43:36,829 [I] DB FLAVOR :POSTGRES
2026-09-24 04:43:36,829 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:43:36,829 [I] Checking connection
2026-09-24 04:43:36,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:43:37,081 [I] connection success
2026-09-24 04:43:37,081 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:43:37,081 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:43:37,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:43:37,327 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:43:37,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:43:37,328 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:43:37,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:43:37,597 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:43:38.093151 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:43:38.306354 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:43:38.360352 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:43:38 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225015, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:38.849421 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:43:38.849573 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhiQnGj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhiQnGj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:43:38.849593 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:43:38.854837 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12568
I20260924 04:43:38.854928 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59397
Sep 24, 2026 4:43:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:43:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:43:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:43:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:43:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:43:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:43:41.103061 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.430s user 0.007s sys 0.149s
I20260924 04:43:41.103202 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:43:41.103288 392 mini_ranger_kms.cc:401] 127.0.98.20:59397/kms/v1/keys
Sep 24 04:43:41 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:43:41 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:43:42 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:43:44 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:43:44.431649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36007
--ranger_config_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:44.565157 12618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:44.565454 12618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:44.565493 12618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:44.569617 12618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:44.569686 12618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:44.569717 12618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:44.569734 12618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:44.569749 12618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:44.576017 12618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36007
--ranger_config_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:44.577181 12618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:44.578289 12618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:44.587509 12624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:44.587730 12618 server_base.cc:1034] running on GCE node
W20260924 04:43:44.587553 12626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:44.587545 12623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:44.588537 12618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:44.589780 12618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:44.590986 12618 hybrid_clock.cc:648] HybridClock initialized: now 1790225024590965 us; error 43 us; skew 500 ppm
Sep 24 04:43:44 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:44.593987 12618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:44.595037 12618 webserver.cc:460] Webserver started at http://127.0.98.62:39695/ using document root <none> and password file <none>
I20260924 04:43:44.595625 12618 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:44.595709 12618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:44.595920 12618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:43:44 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:44.663898 12618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "1121e36c3ef24a73956a8c8def35a133"
format_stamp: "Formatted at 2026-09-24 04:43:44 on dist-test-slave-npg2"
server_key: "1b7f0160c6fc5accd7e5bc43e05d5c24"
server_key_iv: "7eb94fea027fca698931966bee333f34"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:44.664669 12618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "1121e36c3ef24a73956a8c8def35a133"
format_stamp: "Formatted at 2026-09-24 04:43:44 on dist-test-slave-npg2"
server_key: "1b7f0160c6fc5accd7e5bc43e05d5c24"
server_key_iv: "7eb94fea027fca698931966bee333f34"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:44.669651 12618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 04:43:44.687290 12633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:44.689090 12618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:44.689232 12618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "1121e36c3ef24a73956a8c8def35a133"
format_stamp: "Formatted at 2026-09-24 04:43:44 on dist-test-slave-npg2"
server_key: "1b7f0160c6fc5accd7e5bc43e05d5c24"
server_key_iv: "7eb94fea027fca698931966bee333f34"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:44.689371 12618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:44.712988 12618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:44.716459 12618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:44.716677 12618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:44.725385 12618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36007
I20260924 04:43:44.725407 12685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36007 every 8 connection(s)
I20260924 04:43:44.726641 12618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:44.729625 12686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:44.731784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 12618
I20260924 04:43:44.731915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:43:44.732214 392 external_mini_cluster.cc:1428] Setting key 31552b4aecd670e6fdcf9669ca77760e
I20260924 04:43:44.736549 12686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Bootstrap starting.
I20260924 04:43:44.739323 12686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:44.740224 12686 log.cc:824] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:44.742754 12686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: No bootstrap required, opened a new log
Sep 24 04:43:44 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:44.747627 12686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:44.747951 12686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:44.748039 12686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1121e36c3ef24a73956a8c8def35a133, State: Initialized, Role: FOLLOWER
I20260924 04:43:44.748636 12686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:44.748803 12686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:43:44.748888 12686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:43:44.749001 12686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:44.749760 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:44.733740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.0.98.62:36007)
0924 04:43:44.734217 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:44.734235 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:44.734280 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:43:44.735395 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:44.735420 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:44.735866 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:44.736261 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:44.737127 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:44.738614 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:44.739883 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:44.740616 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:44.740788 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:44.743942 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:44.743949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:44.743967 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:44.744001 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:44.746034 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:44.746589 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:44.746597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:44.746604 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:44.746753 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:44.747068 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:44.747071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:44.747073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:44.747639 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:44.747854 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:44.748471 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 04:43:44.748778 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":142,"threads_started":1}
I20260924 04:43:44.750051 12686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:44.750422 12686 leader_election.cc:302] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1121e36c3ef24a73956a8c8def35a133; no voters:
I20260924 04:43:44.750771 12686 leader_election.cc:288] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:43:44.751971 12686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:44.752133 12691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:43:44.752242 12691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:44.752367 12691 leader_election.cc:302] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1121e36c3ef24a73956a8c8def35a133; no voters:
I20260924 04:43:44.752565 12691 leader_election.cc:288] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260924 04:43:44.752668 12691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:44.752739 12691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 LEADER]: Becoming Leader. State: Replica: 1121e36c3ef24a73956a8c8def35a133, State: Running, Role: LEADER
I20260924 04:43:44.753046 12691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:44.754222 12691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1121e36c3ef24a73956a8c8def35a133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } } }
I20260924 04:43:44.754333 12691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:44.754530 12691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1121e36c3ef24a73956a8c8def35a133. Latest consensus state: current_term: 1 leader_uuid: "1121e36c3ef24a73956a8c8def35a133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } } }
I20260924 04:43:44.754598 12691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:44.754572 12686 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:43:44.759418 12706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:44.763895 12706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:44.772181 12706 catalog_manager.cc:1347] Generated new cluster ID: 804fa302e5394874b3f37ce45ca30321
I20260924 04:43:44.772262 12706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:43:44.787707 12706 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:43:44.789127 12706 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:43:44.799917 12706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Generated new TSK 0
I20260924 04:43:44.800787 12706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:43:45.501019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36007
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:45.659092 12732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:45.659458 12732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:45.659502 12732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:45.664332 12732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:45.664422 12732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:45.664510 12732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:43:45.672734 12732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:45.673951 12732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:45.675055 12732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:45.685425 12742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:45.685431 12739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:45.685562 12740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:45.686185 12732 server_base.cc:1034] running on GCE node
I20260924 04:43:45.686717 12732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:45.687510 12732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:45.688688 12732 hybrid_clock.cc:648] HybridClock initialized: now 1790225025688661 us; error 43 us; skew 500 ppm
Sep 24 04:43:45 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:45.691638 12732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:43:45.692605 12732 webserver.cc:460] Webserver started at http://127.0.98.1:39899/ using document root <none> and password file <none>
I20260924 04:43:45.693177 12732 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:45.693238 12732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:45.693466 12732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:43:45 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:45.718747 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "8cd0d9d1e5d346119789079b67e992b4"
format_stamp: "Formatted at 2026-09-24 04:43:45 on dist-test-slave-npg2"
server_key: "c0fc40d73013e2e7f577d12245f2dbc9"
server_key_iv: "6121ad85817d9279b9a82673051d92a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:45.720460 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8cd0d9d1e5d346119789079b67e992b4"
format_stamp: "Formatted at 2026-09-24 04:43:45 on dist-test-slave-npg2"
server_key: "c0fc40d73013e2e7f577d12245f2dbc9"
server_key_iv: "6121ad85817d9279b9a82673051d92a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:45.725965 12732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:43:45.747547 12749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:45.749418 12732 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260924 04:43:45.749627 12732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "8cd0d9d1e5d346119789079b67e992b4"
format_stamp: "Formatted at 2026-09-24 04:43:45 on dist-test-slave-npg2"
server_key: "c0fc40d73013e2e7f577d12245f2dbc9"
server_key_iv: "6121ad85817d9279b9a82673051d92a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:45.749822 12732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:45.796168 12732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:45.800498 12732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:45.800751 12732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:45.801513 12732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:45.802711 12732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:45.802824 12732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:45.803009 12732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:45.803103 12732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:45.814002 12732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45731
I20260924 04:43:45.815498 12732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:43:45.820151 12864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45731 every 8 connection(s)
I20260924 04:43:45.821806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 12732
I20260924 04:43:45.821941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:43:45.822254 392 external_mini_cluster.cc:1428] Setting key ead66afd1a39c8cddf5dfb086fd8f1e3
Sep 24 04:43:45 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:43:45.844486 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:45.823287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36007 (local address 127.0.98.1:52785)
0924 04:43:45.823910 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:45.823971 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:45.828146 (+ 4175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:45.830468 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:45.830479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:45.830894 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:45.831705 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:45.831730 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:45.833137 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:45.833141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:45.834591 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:45.834602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:45.835375 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:45.835381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:45.835598 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:45.836352 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:45.836374 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:45.838209 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:45.840865 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:45.840873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:45.840894 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:45.841293 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:45.841662 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:45.841665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:45.841667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:45.841891 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:45.842381 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:45.842388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:45.842766 (+ 378us) client_negotiation.cc:769] Sending connection context
0924 04:43:45.842978 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 04:43:45.843372 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":169,"threads_started":1}
I20260924 04:43:45.845927 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:45.824518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52785 (local address 127.0.98.62:36007)
0924 04:43:45.829965 (+ 5447us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:45.829970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:45.829988 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:43:45.830031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:45.830034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:45.830087 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:45.831978 (+ 1891us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:45.832107 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:45.832963 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:45.834785 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:45.835190 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:45.838418 (+ 3228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:45.838578 (+ 160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:45.838581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:45.838584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:45.838606 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:45.840698 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:45.841439 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:45.841443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:45.841446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:45.841542 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:45.842021 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:45.842025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:45.842026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:45.842257 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:45.845508 (+ 3251us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:45.845592 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:43:45.845727 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5348,"thread_start_us":97,"threads_started":1}
I20260924 04:43:45.847282 12865 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36007
I20260924 04:43:45.847558 12865 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:45.848187 12865 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:45.850113 12650 ts_manager.cc:194] Registered new tserver with Master: 8cd0d9d1e5d346119789079b67e992b4 (127.0.98.1:45731)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:43:45.851837 12650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52785
Sep 24 04:43:45 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:43:45.879441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36007
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:46.046041 12871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:46.046603 12871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:46.046720 12871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:46.052592 12871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:46.052740 12871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:46.052878 12871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:43:46.062885 12871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:46.064338 12871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:46.065644 12871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:46.080932 12878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:46.081065 12880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:46.082423 12871 server_base.cc:1034] running on GCE node
W20260924 04:43:46.083269 12877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:46.083891 12871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:46.084760 12871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:46.092142 12871 hybrid_clock.cc:648] HybridClock initialized: now 1790225026092057 us; error 95 us; skew 500 ppm
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:46.106294 12871 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:43:46.107606 12871 webserver.cc:460] Webserver started at http://127.0.98.2:46163/ using document root <none> and password file <none>
I20260924 04:43:46.108381 12871 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:46.108464 12871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:46.108717 12871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:46.131445 12871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbf1c68ce34b46a58c8b234fcb0fd747"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0f7d9175b391a0114bb4123e8d9be381"
server_key_iv: "7b218d055c9877ec1ead25952239d46d"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.132356 12871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbf1c68ce34b46a58c8b234fcb0fd747"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0f7d9175b391a0114bb4123e8d9be381"
server_key_iv: "7b218d055c9877ec1ead25952239d46d"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.137017 12871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:43:46.163249 12887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.164978 12871 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260924 04:43:46.165117 12871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "bbf1c68ce34b46a58c8b234fcb0fd747"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0f7d9175b391a0114bb4123e8d9be381"
server_key_iv: "7b218d055c9877ec1ead25952239d46d"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.165261 12871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:46.206126 12871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:46.214830 12871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:46.215039 12871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:46.215785 12871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:46.216864 12871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:46.216926 12871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.216985 12871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:46.217016 12871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.229348 12871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43153
I20260924 04:43:46.230525 12871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:43:46.233982 13000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43153 every 8 connection(s)
I20260924 04:43:46.239451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 12871
I20260924 04:43:46.239583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:43:46.240041 392 external_mini_cluster.cc:1428] Setting key 2557bb5f99bb8a3b619e3814a7b1c9ab
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:46.256326 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:46.234446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48419 (local address 127.0.98.62:36007)
0924 04:43:46.234589 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:46.234594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:46.237143 (+ 2549us) server_negotiation.cc:407] Connection header received
0924 04:43:46.238392 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:46.238396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:46.238455 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:46.238514 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:46.244954 (+ 6440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.245790 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.247746 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.248073 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.248148 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:46.251375 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:46.251379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:46.251382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:46.251414 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:46.253410 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:46.254264 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:46.254267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:46.254269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:46.254357 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:46.254773 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:46.254776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:46.254778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:46.255012 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:46.255092 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:46.255986 (+ 894us) server_negotiation.cc:299] Negotiation successful
0924 04:43:46.256117 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:43:46.258279 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:46.234989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36007 (local address 127.0.98.2:48419)
0924 04:43:46.236970 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:46.237024 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:46.238114 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:46.238939 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:46.238948 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:46.243955 (+ 5007us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:46.244749 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:46.244772 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.246027 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.246030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:46.247605 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:46.247617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.248371 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.248374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:46.248573 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:46.249333 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:46.249354 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:46.251165 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:46.253686 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:46.253693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:46.253714 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:46.254133 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:46.254460 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:46.254463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:46.254465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:46.254643 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:46.255226 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:46.255233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:46.255703 (+ 470us) client_negotiation.cc:769] Sending connection context
0924 04:43:46.256867 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0924 04:43:46.257234 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1746,"thread_start_us":174,"threads_started":1}
I20260924 04:43:46.262646 13001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36007
I20260924 04:43:46.262918 13001 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:46.263674 13001 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:46.264940 12650 ts_manager.cc:194] Registered new tserver with Master: bbf1c68ce34b46a58c8b234fcb0fd747 (127.0.98.2:43153)
I20260924 04:43:46.265684 12650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48419
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:43:46.295868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36007
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:43:46.486753 13006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:46.487553 13006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:46.487644 13006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:46.493904 13006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:46.493988 13006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:46.494072 13006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:43:46.502521 13006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:46.504221 13006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:46.505682 13006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:46.518393 13011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:46.518464 13012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:46.518595 13014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:46.519075 13006 server_base.cc:1034] running on GCE node
I20260924 04:43:46.519563 13006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:46.520390 13006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:46.521576 13006 hybrid_clock.cc:648] HybridClock initialized: now 1790225026521553 us; error 45 us; skew 500 ppm
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:46.525276 13006 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:43:46.526481 13006 webserver.cc:460] Webserver started at http://127.0.98.3:35577/ using document root <none> and password file <none>
I20260924 04:43:46.527351 13006 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:46.527456 13006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:46.527697 13006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:46.546998 13006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "5307776e81f84cc19fef5afbeea3e341"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0b952224f4b59a1f06d2be5a5656dd31"
server_key_iv: "a3c4417b743e8cb73e30ac2d9dc2fac0"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.548302 13006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5307776e81f84cc19fef5afbeea3e341"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0b952224f4b59a1f06d2be5a5656dd31"
server_key_iv: "a3c4417b743e8cb73e30ac2d9dc2fac0"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.554293 13006 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 04:43:46.576341 13021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.578274 13006 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260924 04:43:46.578507 13006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "5307776e81f84cc19fef5afbeea3e341"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0b952224f4b59a1f06d2be5a5656dd31"
server_key_iv: "a3c4417b743e8cb73e30ac2d9dc2fac0"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.578728 13006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:46.596267 13006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:46.600026 13006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:46.600278 13006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:46.600998 13006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:46.602053 13006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:46.602151 13006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.602237 13006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:46.602294 13006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.612715 13006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35277
I20260924 04:43:46.612795 13134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35277 every 8 connection(s)
I20260924 04:43:46.614429 13006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:43:46.616571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13006
I20260924 04:43:46.616705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:43:46.617029 392 external_mini_cluster.cc:1428] Setting key 21bf080ede9fb0352cf894707c7cf71b
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:46.639829 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:46.618529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40875 (local address 127.0.98.62:36007)
0924 04:43:46.618696 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:46.618702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:46.624890 (+ 6188us) server_negotiation.cc:407] Connection header received
0924 04:43:46.626187 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:46.626191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:46.626248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:46.626353 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:46.628409 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.629208 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.631631 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.631948 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.632044 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:46.634946 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:46.634950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:46.634953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:46.634981 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:46.636703 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:46.637698 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:46.637702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:46.637704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:46.637806 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:46.638271 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:46.638275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:46.638276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:46.638514 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:46.638598 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:46.639372 (+ 774us) server_negotiation.cc:299] Negotiation successful
0924 04:43:46.639536 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:43:46.643296 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:46.617297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36007 (local address 127.0.98.3:40875)
0924 04:43:46.624677 (+ 7380us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:46.624752 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:46.625929 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:46.626578 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:46.626589 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:46.626969 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:46.628231 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:46.628256 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.629363 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.629367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:46.631507 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:46.631518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:46.632208 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:46.632212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:46.632419 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:46.633129 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:46.633149 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:46.634752 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:46.636838 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:46.636845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:46.636866 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:46.637260 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:46.637930 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:46.637934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:46.637937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:46.638153 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:46.638667 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:46.638675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:46.639051 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 04:43:46.640082 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0924 04:43:46.640462 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7076,"thread_start_us":160,"threads_started":1}
I20260924 04:43:46.644680 13135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36007
I20260924 04:43:46.644968 13135 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:46.645637 13135 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:46.646888 12650 ts_manager.cc:194] Registered new tserver with Master: 5307776e81f84cc19fef5afbeea3e341 (127.0.98.3:35277)
I20260924 04:43:46.647637 12650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40875
I20260924 04:43:46.654932 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:43:46.655728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 12732
I20260924 04:43:46.664388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 12871
I20260924 04:43:46.670897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13006
I20260924 04:43:46.677434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 12618
I20260924 04:43:46.682911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36007
--webserver_interface=127.0.98.62
--webserver_port=39695
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36007
--ranger_config_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:46.798055 13138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:46.798398 13138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:46.798467 13138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:46.802685 13138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:46.802757 13138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:46.802776 13138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:46.802793 13138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:46.802809 13138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:46.808684 13138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36007
--ranger_config_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:46.809803 13138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:46.810847 13138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:46.820080 13143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:46.820207 13138 server_base.cc:1034] running on GCE node
W20260924 04:43:46.820081 13146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:46.820082 13144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:46.822918 13138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:46.824240 13138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:46.825455 13138 hybrid_clock.cc:648] HybridClock initialized: now 1790225026825422 us; error 52 us; skew 500 ppm
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:46.828795 13138 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:46.829782 13138 webserver.cc:460] Webserver started at http://127.0.98.62:39695/ using document root <none> and password file <none>
I20260924 04:43:46.830399 13138 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:46.830502 13138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:43:46 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:46.887570 13138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:43:46.890218 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:46.891681 13138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:46.891834 13138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "1121e36c3ef24a73956a8c8def35a133"
format_stamp: "Formatted at 2026-09-24 04:43:44 on dist-test-slave-npg2"
server_key: "1b7f0160c6fc5accd7e5bc43e05d5c24"
server_key_iv: "7eb94fea027fca698931966bee333f34"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:46.892436 13138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:46.928960 13138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:46.932417 13138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:46.932669 13138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:46.940393 13138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36007
I20260924 04:43:46.940429 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36007 every 8 connection(s)
I20260924 04:43:46.941522 13138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:46.946031 13207 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:43:46.947685 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Bootstrap starting.
I20260924 04:43:46.951352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13138
I20260924 04:43:46.951900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45731
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36007
--webserver_port=39899
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:46.964793 13207 log.cc:824] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:46.971836 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:43:46.972270 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Bootstrap complete.
I20260924 04:43:46.976692 13207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:46.977095 13207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1121e36c3ef24a73956a8c8def35a133, State: Initialized, Role: FOLLOWER
I20260924 04:43:46.977662 13207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:46.977818 13207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:43:46.977918 13207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:43:46.978044 13207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:43:46.980151 13207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:46.980500 13207 leader_election.cc:302] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1121e36c3ef24a73956a8c8def35a133; no voters:
I20260924 04:43:46.980849 13207 leader_election.cc:288] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:43:46.980947 13211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:43:46.981155 13211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [term 2 LEADER]: Becoming Leader. State: Replica: 1121e36c3ef24a73956a8c8def35a133, State: Running, Role: LEADER
I20260924 04:43:46.981534 13211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } }
I20260924 04:43:46.982128 13207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:46.983017 13213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1121e36c3ef24a73956a8c8def35a133. Latest consensus state: current_term: 2 leader_uuid: "1121e36c3ef24a73956a8c8def35a133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } } }
I20260924 04:43:46.983270 13213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:46.983067 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1121e36c3ef24a73956a8c8def35a133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1121e36c3ef24a73956a8c8def35a133" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36007 } } }
I20260924 04:43:46.983502 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:46.983647 13220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:46.987015 13220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:46.987586 13207 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:43:46.987900 13220 catalog_manager.cc:1259] Loaded cluster ID: 804fa302e5394874b3f37ce45ca30321
I20260924 04:43:46.987942 13220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:43:46.992563 13220 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:43:46.993888 13220 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Loaded TSK: 0
I20260924 04:43:46.995114 13220 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:43:47.101915 13209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:47.102385 13209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:47.102514 13209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:47.108564 13209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:47.108718 13209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:47.108870 13209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:43:47.116935 13209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39899
--tserver_master_addrs=127.0.98.62:36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:47.118507 13209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:47.119859 13209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:47.134218 13245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:47.134388 13209 server_base.cc:1034] running on GCE node
W20260924 04:43:47.134821 13244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:47.135281 13247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:47.135841 13209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:47.136644 13209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:47.137849 13209 hybrid_clock.cc:648] HybridClock initialized: now 1790225027137821 us; error 41 us; skew 500 ppm
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:47.141644 13209 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:43:47.143179 13209 webserver.cc:460] Webserver started at http://127.0.98.1:39899/ using document root <none> and password file <none>
I20260924 04:43:47.143996 13209 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:47.144184 13209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:47.179832 13209 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:43:47.183398 13255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.185108 13209 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:43:47.185240 13209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "8cd0d9d1e5d346119789079b67e992b4"
format_stamp: "Formatted at 2026-09-24 04:43:45 on dist-test-slave-npg2"
server_key: "c0fc40d73013e2e7f577d12245f2dbc9"
server_key_iv: "6121ad85817d9279b9a82673051d92a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:47.186007 13209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:47.211025 13209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:47.214807 13209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:47.215049 13209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:47.215770 13209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:47.216816 13209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:47.216892 13209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.216990 13209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:47.217053 13209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.231140 13209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45731
I20260924 04:43:47.232782 13209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:43:47.232901 13368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45731 every 8 connection(s)
I20260924 04:43:47.243475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13209
I20260924 04:43:47.244050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43153
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36007
--webserver_port=46163
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:47.257195 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.236068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44423 (local address 127.0.98.62:36007)
0924 04:43:47.236571 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:47.236595 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:47.237724 (+ 1129us) server_negotiation.cc:407] Connection header received
0924 04:43:47.239086 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:47.239104 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:47.239608 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:47.240024 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:47.241739 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.243388 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.245217 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.246047 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.246246 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.249602 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:47.249610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:47.249632 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:47.249672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:47.252244 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.253094 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.253101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.253108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.253302 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.253805 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.253809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.253811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.254486 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:47.254648 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:47.255635 (+ 987us) server_negotiation.cc:299] Negotiation successful
0924 04:43:47.255944 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":148,"threads_started":1}
I20260924 04:43:47.259750 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.236290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36007 (local address 127.0.98.1:44423)
0924 04:43:47.237504 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:47.237552 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:47.238540 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:47.240351 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:47.240361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:47.240756 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:47.241506 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:47.241529 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.243579 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.243583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:47.245047 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:47.245062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.246282 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.246286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:47.246487 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.247341 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:47.247367 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:47.249316 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:47.252483 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:47.252491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:47.252509 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:47.252925 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:47.253449 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:47.253455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:47.253457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:47.253666 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:47.254750 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:47.254757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:47.255179 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 04:43:47.258109 (+ 2930us) client_negotiation.cc:240] Negotiation successful
0924 04:43:47.258542 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":146,"threads_started":1}
I20260924 04:43:47.263515 13369 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36007
I20260924 04:43:47.263953 13369 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.264626 13369 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
W20260924 04:43:47.267414 13369 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36007 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:43:47.267637 13369 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.267863 13369 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:47.268800 13369 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.268981 13369 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
W20260924 04:43:47.269829 13369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:43:47.422755 13373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:47.423257 13373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:47.423403 13373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:47.430069 13373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:47.430240 13373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:47.430378 13373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:43:47.439373 13373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46163
--tserver_master_addrs=127.0.98.62:36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:47.441030 13373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:47.442590 13373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:47.456624 13381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:47.456610 13379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:47.457100 13378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:47.457433 13373 server_base.cc:1034] running on GCE node
I20260924 04:43:47.457947 13373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:47.459074 13373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:47.460345 13373 hybrid_clock.cc:648] HybridClock initialized: now 1790225027460291 us; error 65 us; skew 500 ppm
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:47.464643 13373 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:43:47.465981 13373 webserver.cc:460] Webserver started at http://127.0.98.2:46163/ using document root <none> and password file <none>
I20260924 04:43:47.466809 13373 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:47.466969 13373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:47.503543 13373 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.001s
I20260924 04:43:47.507175 13390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.509133 13373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:43:47.509286 13373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "bbf1c68ce34b46a58c8b234fcb0fd747"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0f7d9175b391a0114bb4123e8d9be381"
server_key_iv: "7b218d055c9877ec1ead25952239d46d"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:47.510069 13373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:47.552359 13373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:47.556231 13373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:47.556510 13373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:47.557310 13373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:47.558516 13373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:47.558578 13373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.558635 13373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:47.558660 13373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.570199 13373 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43153
I20260924 04:43:47.571735 13373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:43:47.573376 13503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43153 every 8 connection(s)
I20260924 04:43:47.577517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13373
I20260924 04:43:47.578073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35277
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36007
--webserver_port=35577
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:47.590553 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.575285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45135 (local address 127.0.98.62:36007)
0924 04:43:47.575463 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:47.575466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:47.576326 (+ 860us) server_negotiation.cc:407] Connection header received
0924 04:43:47.577391 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:47.577395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:47.577451 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:47.577571 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:47.579029 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.579790 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.581663 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.582121 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.582273 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.585614 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:47.585620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:47.585627 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:47.585668 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:47.587732 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.588454 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.588460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.588465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.588645 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.589109 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.589116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.589119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.589425 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:47.589558 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:47.590239 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 04:43:47.590376 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:43:47.591789 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.575578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36007 (local address 127.0.98.2:45135)
0924 04:43:47.576139 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:47.576189 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:47.577151 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:47.577939 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:47.577946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:47.578306 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:47.578867 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:47.578882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.579957 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.579960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:47.581494 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:47.581507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.582342 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.582347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:47.582570 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.583411 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:47.583432 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:47.585370 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:47.587887 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:47.587894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:47.587915 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:47.588329 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:47.588769 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:47.588773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:47.588775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:47.588987 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:47.589556 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:47.589564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:47.589932 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 04:43:47.590176 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 04:43:47.590596 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":166,"threads_started":1}
I20260924 04:43:47.593288 13504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36007
I20260924 04:43:47.593554 13504 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.594136 13504 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
W20260924 04:43:47.595986 13504 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36007 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:43:47.596107 13504 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.596288 13504 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:47.597159 13504 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.597370 13504 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
W20260924 04:43:47.598136 13504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:43:47.720724 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:47.721038 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:47.721159 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:47.726506 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:47.726621 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:47.726745 13507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:43:47.734385 13507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35577
--tserver_master_addrs=127.0.98.62:36007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:47.736183 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:47.737638 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:47.750349 13513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:47.750919 13515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:47.751299 13507 server_base.cc:1034] running on GCE node
W20260924 04:43:47.753526 13512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:47.754096 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:47.754945 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:47.756134 13507 hybrid_clock.cc:648] HybridClock initialized: now 1790225027756095 us; error 56 us; skew 500 ppm
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:47.759093 13507 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:43:47.760391 13507 webserver.cc:460] Webserver started at http://127.0.98.3:35577/ using document root <none> and password file <none>
I20260924 04:43:47.761088 13507 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:47.761247 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:47.781136 13507 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:43:47.785369 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.787230 13507 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 04:43:47.787451 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "5307776e81f84cc19fef5afbeea3e341"
format_stamp: "Formatted at 2026-09-24 04:43:46 on dist-test-slave-npg2"
server_key: "0b952224f4b59a1f06d2be5a5656dd31"
server_key_iv: "a3c4417b743e8cb73e30ac2d9dc2fac0"
server_key_version: "kuduclusterkey@0"
I20260924 04:43:47.788362 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:47.815876 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:47.819388 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:47.819595 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:47.820487 13507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:47.821455 13507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:47.821527 13507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.821600 13507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:47.821637 13507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:47.833178 13507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35277
I20260924 04:43:47.833215 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35277 every 8 connection(s)
I20260924 04:43:47.834360 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:43:47.844444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13507
Sep 24 04:43:47 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:47.853610 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.836376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52377 (local address 127.0.98.62:36007)
0924 04:43:47.837770 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:47.837774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:47.837792 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:43:47.838886 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:47.838889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:47.838937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:47.839015 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:47.840655 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.841433 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.843236 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.843625 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.843716 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.846833 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:47.846837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:47.846840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:47.846870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:47.848773 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.851605 (+ 2832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.851609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.851611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.851705 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.852171 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.852175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.852177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.852413 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:47.852498 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:47.853305 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 04:43:47.853438 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304}
I20260924 04:43:47.855803 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.836926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36007 (local address 127.0.98.3:52377)
0924 04:43:47.837487 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:47.837550 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:47.838629 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:47.839282 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:47.839293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:47.839721 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:47.840471 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:47.840497 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.841588 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.841592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:47.843018 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:47.843030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.843950 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.843953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:47.844185 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.844927 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:47.844949 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:47.846627 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:47.850970 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:47.850978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:47.850999 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:47.851472 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:47.851824 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:47.851827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:47.851829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:47.852050 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:47.852566 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:47.852573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:47.852965 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 04:43:47.854138 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0924 04:43:47.854545 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":106,"threads_started":1}
I20260924 04:43:47.857340 13644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36007
I20260924 04:43:47.857683 13644 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:47.858387 13644 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:47.858649 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:47.845494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.98.62:36007)
0924 04:43:47.847307 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:47.847316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:47.847358 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:43:47.847396 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:47.847399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:47.847808 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:47.849766 (+ 1958us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:47.849863 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.850718 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.852364 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:47.852715 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:47.854515 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:47.854674 (+ 159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:47.854678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:47.854681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:47.854700 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:47.856485 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.857071 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.857074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.857076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.857158 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:47.857467 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:47.857471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:47.857472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:47.857716 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:47.857915 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:47.858324 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 04:43:47.858450 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1733,"thread_start_us":118,"threads_started":1}
I20260924 04:43:47.859987 13171 ts_manager.cc:194] Registered new tserver with Master: 5307776e81f84cc19fef5afbeea3e341 (127.0.98.3:35277)
I20260924 04:43:47.861320 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52377
Sep 24 04:43:48 dist-test-slave-npg2 krb5kdc[12174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:48.270107 13369 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:48.270423 13369 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:48.271277 13171 ts_manager.cc:194] Registered new tserver with Master: 8cd0d9d1e5d346119789079b67e992b4 (127.0.98.1:45731)
I20260924 04:43:48.271915 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44423
Sep 24 04:43:48 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:48.598364 13504 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:48.598690 13504 heartbeater.cc:511] Master 127.0.98.62:36007 requested a full tablet report, sending...
I20260924 04:43:48.599509 13171 ts_manager.cc:194] Registered new tserver with Master: bbf1c68ce34b46a58c8b234fcb0fd747 (127.0.98.2:43153)
I20260924 04:43:48.600245 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45135
I20260924 04:43:48.607914 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:43:48.625705 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.612584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.98.62:36007)
0924 04:43:48.614597 (+ 2013us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:48.614604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:48.615010 (+ 406us) server_negotiation.cc:407] Connection header received
0924 04:43:48.615067 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:48.615069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:48.615157 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:48.615248 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:48.616319 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.617200 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.618391 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.618716 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.618829 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.621151 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:48.621155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:48.621158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:48.621187 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:48.623055 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:48.623619 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:48.623622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:48.623624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:48.623733 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:48.624074 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:48.624077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:48.624079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:48.624314 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:48.624442 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:48.625323 (+ 881us) server_negotiation.cc:299] Negotiation successful
0924 04:43:48.625472 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1894,"thread_start_us":106,"threads_started":1}
I20260924 04:43:48.630764 13171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:43:48.641657 13171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:43:48.674218 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.663644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43153 (local address 127.0.0.1:45520)
0924 04:43:48.665542 (+ 1898us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:48.665600 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:48.665743 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:48.666371 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:48.666382 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:48.666407 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:48.666569 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:48.666578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.667975 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.667978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:48.669352 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:48.669363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.673433 (+ 4070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.673440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:48.673632 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.673700 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 04:43:48.673820 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 04:43:48.674027 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"mutex_wait_us":1,"thread_start_us":133,"threads_started":1}
I20260924 04:43:48.676360 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.664732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.0.98.2:43153)
0924 04:43:48.665278 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:48.665285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:48.665888 (+ 603us) server_negotiation.cc:407] Connection header received
0924 04:43:48.665973 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:48.665978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:48.666226 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:48.666391 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:48.666723 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.667854 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.670831 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.673031 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.673165 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.673347 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:48.676065 (+ 2718us) server_negotiation.cc:299] Negotiation successful
0924 04:43:48.676193 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":101,"threads_started":1}
I20260924 04:43:48.680660 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.664838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45731 (local address 127.0.0.1:37032)
0924 04:43:48.667784 (+ 2946us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:48.667797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:48.667919 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:48.676120 (+ 8201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:48.676124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:48.676155 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:48.676250 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:48.676256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.677629 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.677632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:48.678814 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:48.678825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.680205 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.680209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:48.680327 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.680357 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:43:48.680435 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:43:48.680512 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"thread_start_us":83,"threads_started":1}
I20260924 04:43:48.680819 13438 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3f1648c143419c9b4cbbba79bce66f (DEFAULT_TABLE table=test-table [id=157b9d368f8947daa4d0f13230b50991]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:48.681428 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.665028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.0.98.1:45731)
0924 04:43:48.675379 (+ 10351us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:48.675387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:48.675421 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:43:48.675500 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:48.675506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:48.675763 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:48.675951 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:48.676385 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.677470 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.678980 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.679843 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.679951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.680106 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:48.681173 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 04:43:48.681281 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10206,"thread_start_us":128,"threads_started":1}
I20260924 04:43:48.683409 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3f1648c143419c9b4cbbba79bce66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:48.684612 13302 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3f1648c143419c9b4cbbba79bce66f (DEFAULT_TABLE table=test-table [id=157b9d368f8947daa4d0f13230b50991]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:48.685751 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3f1648c143419c9b4cbbba79bce66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:48.695007 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.665107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.0.98.3:35277)
0924 04:43:48.665463 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:48.665469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:48.666839 (+ 1370us) server_negotiation.cc:407] Connection header received
0924 04:43:48.666918 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:48.666923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:48.667116 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:48.667555 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:48.668228 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.669548 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.692895 (+ 23347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.693898 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.694002 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.694204 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:48.694733 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 04:43:48.694850 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":83,"threads_started":1}
I20260924 04:43:48.695566 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:48.663733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35277 (local address 127.0.0.1:57588)
0924 04:43:48.666648 (+ 2915us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:48.666663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:48.666744 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:48.668009 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:48.668012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:48.668027 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:48.668101 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:48.668106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.673722 (+ 5616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.673725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:48.692684 (+ 18959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:48.692702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:48.694401 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:48.694405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:48.694538 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:48.694566 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:43:48.695296 (+ 730us) client_negotiation.cc:240] Negotiation successful
0924 04:43:48.695410 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2833,"mutex_wait_us":1968,"thread_start_us":77,"threads_started":1}
I20260924 04:43:48.698665 13670 tablet_bootstrap.cc:492] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Bootstrap starting.
I20260924 04:43:48.698910 13578 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3f1648c143419c9b4cbbba79bce66f (DEFAULT_TABLE table=test-table [id=157b9d368f8947daa4d0f13230b50991]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:48.700095 13578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3f1648c143419c9b4cbbba79bce66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:48.701200 13670 tablet_bootstrap.cc:654] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:48.702024 13670 log.cc:824] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:48.706888 13671 tablet_bootstrap.cc:492] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: Bootstrap starting.
I20260924 04:43:48.709429 13671 tablet_bootstrap.cc:654] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:48.710359 13671 log.cc:824] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:48.710767 13673 tablet_bootstrap.cc:492] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Bootstrap starting.
I20260924 04:43:48.712930 13670 tablet_bootstrap.cc:492] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: No bootstrap required, opened a new log
I20260924 04:43:48.713176 13670 ts_tablet_manager.cc:1397] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260924 04:43:48.713269 13673 tablet_bootstrap.cc:654] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:48.715448 13673 log.cc:824] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:48.717837 13673 tablet_bootstrap.cc:492] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: No bootstrap required, opened a new log
I20260924 04:43:48.717635 13670 raft_consensus.cc:348] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:48.717939 13670 raft_consensus.cc:374] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:48.718000 13670 raft_consensus.cc:729] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf1c68ce34b46a58c8b234fcb0fd747, State: Initialized, Role: FOLLOWER
I20260924 04:43:48.718050 13673 ts_tablet_manager.cc:1397] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:43:48.718528 13670 consensus_queue.cc:260] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:48.719652 13670 ts_tablet_manager.cc:1428] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260924 04:43:48.720028 13504 heartbeater.cc:503] Master 127.0.98.62:36007 was elected leader, sending a full tablet report...
I20260924 04:43:48.720228 13671 tablet_bootstrap.cc:492] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: No bootstrap required, opened a new log
I20260924 04:43:48.720420 13671 ts_tablet_manager.cc:1397] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260924 04:43:48.724651 13671 raft_consensus.cc:348] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:48.725021 13671 raft_consensus.cc:374] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:48.725129 13671 raft_consensus.cc:729] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd0d9d1e5d346119789079b67e992b4, State: Initialized, Role: FOLLOWER
I20260924 04:43:48.725694 13671 consensus_queue.cc:260] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:48.726827 13671 ts_tablet_manager.cc:1428] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:43:48.727250 13369 heartbeater.cc:503] Master 127.0.98.62:36007 was elected leader, sending a full tablet report...
I20260924 04:43:48.727800 13673 raft_consensus.cc:348] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:48.728075 13673 raft_consensus.cc:374] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:48.728127 13673 raft_consensus.cc:729] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5307776e81f84cc19fef5afbeea3e341, State: Initialized, Role: FOLLOWER
I20260924 04:43:48.728648 13673 consensus_queue.cc:260] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:48.729708 13673 ts_tablet_manager.cc:1428] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20260924 04:43:48.730098 13644 heartbeater.cc:503] Master 127.0.98.62:36007 was elected leader, sending a full tablet report...
W20260924 04:43:48.734995 13370 tablet.cc:2369] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:43:48.824518 13505 tablet.cc:2369] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:43:48.836139 13645 tablet.cc:2369] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:43:48 dist-test-slave-npg2 krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:43:49.119616 13676 raft_consensus.cc:482] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:43:49.119866 13676 raft_consensus.cc:504] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:49.121323 13676 leader_election.cc:288] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5307776e81f84cc19fef5afbeea3e341 (127.0.98.3:35277), 8cd0d9d1e5d346119789079b67e992b4 (127.0.98.1:45731)
I20260924 04:43:49.137557 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.122777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59401 (local address 127.0.98.3:35277)
0924 04:43:49.123055 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:49.123061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:49.127481 (+ 4420us) server_negotiation.cc:407] Connection header received
0924 04:43:49.128491 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:49.128495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:49.128569 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:49.128627 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:49.129796 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.130583 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.134862 (+ 4279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.135517 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.135622 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:49.135701 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:49.137286 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0924 04:43:49.137386 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 04:43:49.138317 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.121806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35277 (local address 127.0.98.2:59401)
0924 04:43:49.127362 (+ 5556us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:49.127382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:49.128065 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:49.128792 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:49.128795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:49.128814 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:49.128967 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:49.128974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.130740 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.130743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:49.132210 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:49.132221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.135808 (+ 3587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.135812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:49.135931 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:49.135955 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:43:49.138042 (+ 2087us) client_negotiation.cc:240] Negotiation successful
0924 04:43:49.138135 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5456,"thread_start_us":120,"threads_started":1}
I20260924 04:43:49.138836 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.122901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42595 (local address 127.0.98.1:45731)
0924 04:43:49.123174 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:49.123178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:49.128389 (+ 5211us) server_negotiation.cc:407] Connection header received
0924 04:43:49.129274 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:49.129278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:49.129333 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:49.129381 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:49.132392 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.133239 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.136114 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.136759 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.136847 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:49.136911 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:49.138614 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0924 04:43:49.138710 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260924 04:43:49.140213 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3f1648c143419c9b4cbbba79bce66f" candidate_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5307776e81f84cc19fef5afbeea3e341" is_pre_election: true
I20260924 04:43:49.140602 13598 raft_consensus.cc:2454] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf1c68ce34b46a58c8b234fcb0fd747 in term 0.
I20260924 04:43:49.141299 13391 leader_election.cc:302] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5307776e81f84cc19fef5afbeea3e341, bbf1c68ce34b46a58c8b234fcb0fd747; no voters:
I20260924 04:43:49.141661 13676 raft_consensus.cc:2781] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:43:49.141739 13676 raft_consensus.cc:482] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:43:49.141782 13676 raft_consensus.cc:3037] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:49.143224 13676 raft_consensus.cc:504] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:49.143643 13676 leader_election.cc:288] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [CANDIDATE]: Term 1 election: Requested vote from peers 5307776e81f84cc19fef5afbeea3e341 (127.0.98.3:35277), 8cd0d9d1e5d346119789079b67e992b4 (127.0.98.1:45731)
I20260924 04:43:49.144309 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3f1648c143419c9b4cbbba79bce66f" candidate_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5307776e81f84cc19fef5afbeea3e341"
I20260924 04:43:49.144443 13598 raft_consensus.cc:3037] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:49.145920 13598 raft_consensus.cc:2454] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf1c68ce34b46a58c8b234fcb0fd747 in term 1.
I20260924 04:43:49.139488 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.122224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45731 (local address 127.0.98.2:42595)
0924 04:43:49.128303 (+ 6079us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:49.128317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:49.129118 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:49.129511 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:49.129514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:49.129542 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:49.129663 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:49.129668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.133394 (+ 3726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.133397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:49.134697 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:49.134708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.137010 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.137014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:49.137128 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:49.137150 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:43:49.139270 (+ 2120us) client_negotiation.cc:240] Negotiation successful
0924 04:43:49.139350 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6013,"thread_start_us":86,"threads_started":1}
I20260924 04:43:49.146344 13391 leader_election.cc:302] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5307776e81f84cc19fef5afbeea3e341, bbf1c68ce34b46a58c8b234fcb0fd747; no voters:
I20260924 04:43:49.147013 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3f1648c143419c9b4cbbba79bce66f" candidate_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd0d9d1e5d346119789079b67e992b4"
I20260924 04:43:49.147044 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3f1648c143419c9b4cbbba79bce66f" candidate_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd0d9d1e5d346119789079b67e992b4" is_pre_election: true
I20260924 04:43:49.147346 13323 raft_consensus.cc:2454] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf1c68ce34b46a58c8b234fcb0fd747 in term 0.
I20260924 04:43:49.147837 13676 raft_consensus.cc:2781] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:49.148337 13676 raft_consensus.cc:686] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 1 LEADER]: Becoming Leader. State: Replica: bbf1c68ce34b46a58c8b234fcb0fd747, State: Running, Role: LEADER
I20260924 04:43:49.148696 13676 consensus_queue.cc:237] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } }
I20260924 04:43:49.152299 13171 catalog_manager.cc:5495] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf1c68ce34b46a58c8b234fcb0fd747 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf1c68ce34b46a58c8b234fcb0fd747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 } health_report { overall_health: UNKNOWN } } }
I20260924 04:43:49.242803 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.239076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.0.98.2:43153)
0924 04:43:49.239381 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:49.239386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:49.239408 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:43:49.239537 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:49.239539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:49.239595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:49.239694 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:43:49.239940 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.240446 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.241328 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:49.241564 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:49.241646 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:49.241990 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:43:49.242452 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:43:49.242551 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:49.242637 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:43:49.242688 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260924 04:43:49.249166 13322 raft_consensus.cc:3037] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:49.249375 13598 raft_consensus.cc:1261] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 1 FOLLOWER]: Refusing update from remote peer bbf1c68ce34b46a58c8b234fcb0fd747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:49.250187 13690 consensus_queue.cc:1035] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5307776e81f84cc19fef5afbeea3e341" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:49.250810 13322 raft_consensus.cc:1261] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 1 FOLLOWER]: Refusing update from remote peer bbf1c68ce34b46a58c8b234fcb0fd747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:49.251467 13690 consensus_queue.cc:1035] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd0d9d1e5d346119789079b67e992b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:49.258253 13695 mvcc.cc:205] Tried to move back new op lower bound from 7332761719795630080 to 7332761719396290560. Current Snapshot: MvccSnapshot[applied={T|T < 7332761719795630080}]
I20260924 04:43:49.259617 13698 mvcc.cc:205] Tried to move back new op lower bound from 7332761719795630080 to 7332761719396290560. Current Snapshot: MvccSnapshot[applied={T|T < 7332761719795630080}]
I20260924 04:43:49.261564 13699 mvcc.cc:205] Tried to move back new op lower bound from 7332761719795630080 to 7332761719396290560. Current Snapshot: MvccSnapshot[applied={T|T < 7332761719795630080}]
I20260924 04:43:49.267802 13170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:43:49.267961 13170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:43:49.270411 13170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1121e36c3ef24a73956a8c8def35a133: Sending DeleteTablet for 3 replicas of tablet 2e3f1648c143419c9b4cbbba79bce66f
I20260924 04:43:49.271294 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3f1648c143419c9b4cbbba79bce66f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45520
I20260924 04:43:49.271397 13302 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3f1648c143419c9b4cbbba79bce66f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37032
I20260924 04:43:49.271402 13578 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3f1648c143419c9b4cbbba79bce66f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57588
I20260924 04:43:49.271806 13708 tablet_replica.cc:330] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4: stopping tablet replica
I20260924 04:43:49.271936 13707 tablet_replica.cc:330] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: stopping tablet replica
I20260924 04:43:49.272169 13709 tablet_replica.cc:330] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: stopping tablet replica
I20260924 04:43:49.272171 13708 raft_consensus.cc:2229] T 2e3f1648c143419c9b4cbbba79bce66f P 8cd0d9d1e5d346119789079b67e992b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:49.272508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13209
I20260924 04:43:49.272512 13707 raft_consensus.cc:2229] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:43:49.274426 13709 raft_consensus.cc:2229] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:49.274794 13709 raft_consensus.cc:2258] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:49.275380 13707 raft_consensus.cc:2258] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:49.276503 13709 ts_tablet_manager.cc:1905] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:43:49.279223 13709 ts_tablet_manager.cc:1918] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:43:49.279423 13709 log.cc:1196] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/2e3f1648c143419c9b4cbbba79bce66f
I20260924 04:43:49.279892 13709 ts_tablet_manager.cc:1939] T 2e3f1648c143419c9b4cbbba79bce66f P 5307776e81f84cc19fef5afbeea3e341: Deleting consensus metadata
I20260924 04:43:49.280045 13707 ts_tablet_manager.cc:1905] T 2e3f1648c143419c9b4cbbba79bce66f P bbf1c68ce34b46a58c8b234fcb0fd747: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:43:49.280778 13157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45731 (error 108)
I20260924 04:43:49.281483 13155 catalog_manager.cc:4841] TS 5307776e81f84cc19fef5afbeea3e341 (127.0.98.3:35277): tablet 2e3f1648c143419c9b4cbbba79bce66f (table test-table [id=157b9d368f8947daa4d0f13230b50991]) successfully deleted
I20260924 04:43:49.282196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13373
I20260924 04:43:49.284042 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.282248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45731 (local address 127.0.0.1:37036)
0924 04:43:49.283839 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:49.283904 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1457,"thread_start_us":66,"threads_started":1}
W20260924 04:43:49.284235 13157 catalog_manager.cc:4584] TS 8cd0d9d1e5d346119789079b67e992b4 (127.0.98.1:45731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e3f1648c143419c9b4cbbba79bce66f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45731: connect: Connection refused (error 111)
I20260924 04:43:49.288355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13507
I20260924 04:43:49.289924 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:49.288545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43153 (local address 127.0.0.1:45536)
0924 04:43:49.289801 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:49.289850 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1171}
I20260924 04:43:49.294139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13138
I20260924 04:43:49.301592 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:43:49.647388 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:43:49.647505 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:43:49.993186 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:43:49.993 UTC [12204] LOG: received smart shutdown request
2026-09-24 04:43:49.997 UTC [12204] LOG: background worker "logical replication launcher" (PID 12213) exited with exit code 1
2026-09-24 04:43:49.998 UTC [12208] LOG: shutting down
2026-09-24 04:43:50.212 UTC [12204] LOG: database system is shut down
2026-09-24T04:43:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51199 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:43:50 dist-test-slave-npg2 krb5kdc[13720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:43:50 dist-test-slave-npg2 krb5kdc[13720](info): set up 2 sockets
Sep 24 04:43:50 dist-test-slave-npg2 krb5kdc[13720](info): commencing operation
krb5kdc: starting...
W20260924 04:43:51.395505 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:43:51.526095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39211,127.0.98.61:46629,127.0.98.60:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:51.639431 13733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:51.639786 13733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:51.639868 13733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:51.644186 13733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:51.644302 13733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:51.644338 13733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:51.644405 13733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:51.644443 13733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:51.650236 13733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39211,127.0.98.61:46629,127.0.98.60:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:51.651484 13733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:51.652549 13733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:51.661295 13738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:51.661415 13741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:51.661513 13739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:51.661562 13733 server_base.cc:1034] running on GCE node
I20260924 04:43:51.662202 13733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:51.663447 13733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:51.664654 13733 hybrid_clock.cc:648] HybridClock initialized: now 1790225031664640 us; error 29 us; skew 500 ppm
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:51.674957 13733 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:51.676128 13733 webserver.cc:460] Webserver started at http://127.0.98.62:40391/ using document root <none> and password file <none>
I20260924 04:43:51.676738 13733 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:51.676831 13733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:51.677073 13733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:51.679312 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "3456e280e95f4b92a41ba281f6cdf536"
format_stamp: "Formatted at 2026-09-24 04:43:51 on dist-test-slave-npg2"
server_key: "1107d4b136834a39018f6c6bdba4d1f9"
server_key_iv: "5d20078b4d3cd4b751a7f1736e9c7b74"
server_key_version: "encryptionkey@0"
I20260924 04:43:51.679889 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "3456e280e95f4b92a41ba281f6cdf536"
format_stamp: "Formatted at 2026-09-24 04:43:51 on dist-test-slave-npg2"
server_key: "1107d4b136834a39018f6c6bdba4d1f9"
server_key_iv: "5d20078b4d3cd4b751a7f1736e9c7b74"
server_key_version: "encryptionkey@0"
I20260924 04:43:51.683831 13733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:43:51.686362 13748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:51.687729 13733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:43:51.687870 13733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "3456e280e95f4b92a41ba281f6cdf536"
format_stamp: "Formatted at 2026-09-24 04:43:51 on dist-test-slave-npg2"
server_key: "1107d4b136834a39018f6c6bdba4d1f9"
server_key_iv: "5d20078b4d3cd4b751a7f1736e9c7b74"
server_key_version: "encryptionkey@0"
I20260924 04:43:51.687990 13733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:51.707046 13733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:51.710381 13733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:51.710605 13733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:51.717998 13733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39211
I20260924 04:43:51.718014 13800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39211 every 8 connection(s)
I20260924 04:43:51.719046 13733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:51.722016 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:51.722997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13733
I20260924 04:43:51.723177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:43:51.723531 392 external_mini_cluster.cc:1428] Setting key 3b2dfe9b1ca960132ba54641f18efbd3
I20260924 04:43:51.727113 13801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:51.743641 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.729833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.0.98.62:39211)
0924 04:43:51.730286 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:51.730305 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:51.730593 (+ 288us) server_negotiation.cc:407] Connection header received
0924 04:43:51.731765 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:51.731769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:51.731958 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:51.732153 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:51.733058 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.734065 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.735296 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.735766 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.735973 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.738558 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:51.738564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:51.738574 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:51.738594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:51.740411 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.740907 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.740913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.740915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.741021 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.741339 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.741342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.741344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.741721 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:51.741880 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:51.742233 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 04:43:51.742486 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260924 04:43:51.743639 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.729834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39211 (local address 127.0.0.1:38744)
0924 04:43:51.730445 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.730483 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:51.731453 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:51.732148 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:51.732154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:51.732614 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:51.732908 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.732922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.734189 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.734192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.735102 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.735115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.735973 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.735977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.736127 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.736852 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:51.736870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:51.738392 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:51.740543 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.740553 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.740564 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.740786 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.741117 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.741122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.741124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.741235 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.741840 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:51.741846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:51.741995 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 04:43:51.742164 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 04:43:51.742538 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":112,"thread_start_us":158,"threads_started":1}
I20260924 04:43:51.746546 13801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:51.747555 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.747210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41534)
0924 04:43:51.747357 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.747474 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:43:51.747762 13749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
I20260924 04:43:51.748741 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.748520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41538)
0924 04:43:51.748640 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.748684 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 04:43:51.748896 13801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:43:51.750597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46629
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39211,127.0.98.61:46629,127.0.98.60:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:51.798221 13801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } attempt: 1
I20260924 04:43:51.799253 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.798896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41546)
0924 04:43:51.799058 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.799173 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 04:43:51.800004 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.799786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41560)
0924 04:43:51.799899 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.799952 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:43:51.800137 13801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
W20260924 04:43:51.866036 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:51.866360 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:51.866467 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:43:51.868332 13801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } attempt: 2
I20260924 04:43:51.869227 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.868910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41562)
0924 04:43:51.869068 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.869147 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 04:43:51.870033 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.869777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41564)
0924 04:43:51.869918 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.869974 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 04:43:51.870173 13801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46629: connect: Connection refused (error 111)
W20260924 04:43:51.870873 13808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:51.870951 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:51.871013 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:51.871064 13808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:51.871102 13808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:51.876634 13808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39211,127.0.98.61:46629,127.0.98.60:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:51.877792 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:51.878824 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:51.887970 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:51.888038 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:51.887974 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:51.888545 13808 server_base.cc:1034] running on GCE node
I20260924 04:43:51.889113 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:51.890268 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:51.891431 13808 hybrid_clock.cc:648] HybridClock initialized: now 1790225031891416 us; error 31 us; skew 500 ppm
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:51.894523 13808 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:43:51.895624 13808 webserver.cc:460] Webserver started at http://127.0.98.61:40817/ using document root <none> and password file <none>
I20260924 04:43:51.896209 13808 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:51.896279 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:51.896504 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:51.898658 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data/instance:
uuid: "f038525da28a4b3186d184065d0604ae"
format_stamp: "Formatted at 2026-09-24 04:43:51 on dist-test-slave-npg2"
server_key: "3a4c4a6396c949a72ea95fd4e46c3bf3"
server_key_iv: "75ee58fe8ebed24e45964b7a7ceded92"
server_key_version: "encryptionkey@0"
I20260924 04:43:51.899343 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal/instance:
uuid: "f038525da28a4b3186d184065d0604ae"
format_stamp: "Formatted at 2026-09-24 04:43:51 on dist-test-slave-npg2"
server_key: "3a4c4a6396c949a72ea95fd4e46c3bf3"
server_key_iv: "75ee58fe8ebed24e45964b7a7ceded92"
server_key_version: "encryptionkey@0"
I20260924 04:43:51.903170 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:43:51.905840 13823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:51.907296 13808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:51.907420 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal
uuid: "f038525da28a4b3186d184065d0604ae"
format_stamp: "Formatted at 2026-09-24 04:43:51 on dist-test-slave-npg2"
server_key: "3a4c4a6396c949a72ea95fd4e46c3bf3"
server_key_iv: "75ee58fe8ebed24e45964b7a7ceded92"
server_key_version: "encryptionkey@0"
I20260924 04:43:51.907539 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:51.929701 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:51.933495 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:51.933732 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:51.941838 13808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46629
I20260924 04:43:51.941840 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46629 every 8 connection(s)
I20260924 04:43:51.942957 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:43:51.946128 13876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:51.947785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13808
I20260924 04:43:51.947928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-1/wal/instance
I20260924 04:43:51.948241 392 external_mini_cluster.cc:1428] Setting key 10666049bce3638d048375fece4611d9
I20260924 04:43:51.951108 13876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 04:43:51.961349 13801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } attempt: 3
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:51.967187 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.953232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.98.62:39211)
0924 04:43:51.953372 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:51.953376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:51.954623 (+ 1247us) server_negotiation.cc:407] Connection header received
0924 04:43:51.955789 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:51.955793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:51.955857 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:51.955963 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:51.957261 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.957783 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.959174 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.959508 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.959639 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.962562 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:51.962569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:51.962575 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:51.962608 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:51.964439 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.965113 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.965118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.965123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.965223 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.965637 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.965643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.965645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.965888 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:51.965995 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:51.966871 (+ 876us) server_negotiation.cc:299] Negotiation successful
0924 04:43:51.967009 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:43:51.967921 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.953823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39211 (local address 127.0.0.1:38746)
0924 04:43:51.954441 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.954488 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:51.955558 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:51.956143 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:51.956150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:51.956543 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:51.957075 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.957089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.957936 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.957939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.959026 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.959036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.959665 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.959668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.959820 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.960452 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:51.960512 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:51.962374 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:51.964583 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.964590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.964610 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.965003 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.965333 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.965336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.965338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.965522 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.966003 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:51.966008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:51.966347 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 04:43:51.966570 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 04:43:51.966934 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"spinlock_wait_cycles":896,"thread_start_us":167,"threads_started":1}
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:43:51.969456 13876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:51.972543 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.961872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41580)
0924 04:43:51.962051 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.962067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:51.962146 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:51.962809 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:51.962813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:51.963006 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:51.963175 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.963183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.964834 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.964837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.965621 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.965631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.966277 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.966283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.966415 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.966844 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:51.966861 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:51.968713 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:51.970866 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.970869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.970872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.971215 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.971513 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.971516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.971518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.971613 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.972069 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:51.972076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:51.972195 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 04:43:51.972305 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:43:51.972424 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 04:43:51.972759 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.961974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.0.98.61:46629)
0924 04:43:51.962398 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:51.962422 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:51.962446 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:43:51.962490 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:51.962497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:51.962682 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:51.962823 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:51.963287 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.964296 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.965725 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.966154 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.966255 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.968854 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:51.968865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:51.968883 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:51.968904 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:51.970744 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.971336 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.971341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.971342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.971418 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.971715 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.971717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.971718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.971949 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:51.972186 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:51.972413 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 04:43:51.972556 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":132,"threads_started":1}
Sep 24 04:43:51 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:43:51.975208 13801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:51.975980 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.975705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35224)
0924 04:43:51.975824 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.975896 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:43:51.976855 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.976561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35226)
0924 04:43:51.976707 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.976784 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 04:43:51.977002 13801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:43:51.978778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43085
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39211,127.0.98.61:46629,127.0.98.60:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:51.979539 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.970062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41584)
0924 04:43:51.970159 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.970170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:51.970254 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:51.970778 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:51.970781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:51.970943 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:51.971058 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.971070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.972211 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.972214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.973010 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:51.973022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.973506 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.973510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:51.973601 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.974113 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:51.974127 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:51.975423 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:51.977881 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.977884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.977892 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.978122 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.978402 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:51.978405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:51.978407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:51.978505 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:51.979002 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:51.979005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:51.979088 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:43:51.979245 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:43:51.979382 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 04:43:51.979815 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.970239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.0.98.61:46629)
0924 04:43:51.970565 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:51.970577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:51.970592 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:43:51.970633 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:51.970636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:51.970670 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:51.970770 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:51.971225 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.971983 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.973129 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:51.973396 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:51.973478 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:51.975595 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:51.975602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:51.975608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:51.975636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:51.977734 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.978227 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.978230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.978231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.978309 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:51.978623 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:51.978626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:51.978627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:51.978829 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:51.978979 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:51.979298 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 04:43:51.979506 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260924 04:43:51.980423 13876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:51.981153 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.980908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35232)
0924 04:43:51.981018 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.981094 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:43:51.981456 13826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
I20260924 04:43:51.982265 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:51.982100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35236)
0924 04:43:51.982175 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:51.982218 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 04:43:51.982497 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
I20260924 04:43:52.008183 13801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } attempt: 1
I20260924 04:43:52.009142 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.008772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35238)
0924 04:43:52.008961 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.009040 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 04:43:52.010133 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.009830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35248)
0924 04:43:52.009981 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.010054 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 04:43:52.010305 13801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
I20260924 04:43:52.031754 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } attempt: 1
I20260924 04:43:52.032733 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.032414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35264)
0924 04:43:52.032584 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.032652 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 04:43:52.033595 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.033268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35274)
0924 04:43:52.033476 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.033541 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260924 04:43:52.033828 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
I20260924 04:43:52.085578 13801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } attempt: 2
I20260924 04:43:52.086496 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.086137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35290)
0924 04:43:52.086314 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.086396 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 04:43:52.087419 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.087113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35294)
0924 04:43:52.087289 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.087351 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:43:52.087634 13801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
W20260924 04:43:52.097853 13883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:52.098132 13883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:52.098212 13883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:43:52.102022 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } attempt: 2
W20260924 04:43:52.102517 13883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:52.102595 13883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:52.102629 13883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:52.102686 13883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:52.102722 13883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:52.102867 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.102584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35308)
0924 04:43:52.102722 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.102792 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:43:52.103627 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.103400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35324)
0924 04:43:52.103510 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.103571 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:43:52.103758 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43085: connect: Connection refused (error 111)
I20260924 04:43:52.108618 13883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39211,127.0.98.61:46629,127.0.98.60:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:52.109824 13883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:52.110960 13883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:52.119771 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:52.119951 13883 server_base.cc:1034] running on GCE node
W20260924 04:43:52.119971 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:52.119802 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:52.120723 13883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:52.121999 13883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:52.123224 13883 hybrid_clock.cc:648] HybridClock initialized: now 1790225032123198 us; error 50 us; skew 500 ppm
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:52.126256 13883 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:43:52.127307 13883 webserver.cc:460] Webserver started at http://127.0.98.60:43243/ using document root <none> and password file <none>
I20260924 04:43:52.127890 13883 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:52.127962 13883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:52.128196 13883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:52.130280 13883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data/instance:
uuid: "46adede31efb4289a14ea5443475b66f"
format_stamp: "Formatted at 2026-09-24 04:43:52 on dist-test-slave-npg2"
server_key: "5427aff6766db0b60a59c3856344214e"
server_key_iv: "d8402968deb66f27130256688129bfca"
server_key_version: "encryptionkey@0"
I20260924 04:43:52.130846 13883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal/instance:
uuid: "46adede31efb4289a14ea5443475b66f"
format_stamp: "Formatted at 2026-09-24 04:43:52 on dist-test-slave-npg2"
server_key: "5427aff6766db0b60a59c3856344214e"
server_key_iv: "d8402968deb66f27130256688129bfca"
server_key_version: "encryptionkey@0"
I20260924 04:43:52.135054 13883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:43:52.137676 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:52.139009 13883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:43:52.139194 13883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal
uuid: "46adede31efb4289a14ea5443475b66f"
format_stamp: "Formatted at 2026-09-24 04:43:52 on dist-test-slave-npg2"
server_key: "5427aff6766db0b60a59c3856344214e"
server_key_iv: "d8402968deb66f27130256688129bfca"
server_key_version: "encryptionkey@0"
I20260924 04:43:52.139299 13883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:52.172602 13883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:52.176102 13883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:52.176268 13883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:52.183094 13883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43085
I20260924 04:43:52.183218 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43085 every 8 connection(s)
I20260924 04:43:52.184140 13883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:43:52.186398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 13883
I20260924 04:43:52.186579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224865297037-392-0/minicluster-data/master-2/wal/instance
I20260924 04:43:52.186902 392 external_mini_cluster.cc:1428] Setting key 7e0d85dc5c479a9c2073e9af496e0b64
I20260924 04:43:52.186857 13801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } attempt: 3
I20260924 04:43:52.187242 13952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:52.192499 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:52.195195 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } attempt: 3
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:43:52.201177 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.187434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35338)
0924 04:43:52.187650 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.187667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:52.187760 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:52.190242 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:52.190246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:52.190464 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:52.190591 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.190599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.192347 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.192350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.193135 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.193147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.193962 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.193966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.194050 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.194743 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:52.194757 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:52.196329 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:52.199032 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.199036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.199038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.199329 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.199708 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.199712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.199713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.199809 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.200678 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:52.200682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:52.200776 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:43:52.200893 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:43:52.201048 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:52.203002 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.188174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.98.60:43085)
0924 04:43:52.188687 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.188696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.188726 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:43:52.189524 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.189548 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.189924 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.190713 (+ 789us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.190827 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.192210 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.193283 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.193826 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.194350 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.196522 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.196529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.196541 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.196572 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.198905 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.199449 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.199454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.199460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.199598 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.199952 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.199955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.199957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.200547 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.201414 (+ 867us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.201602 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.201902 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":282,"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":1792,"thread_start_us":103,"threads_started":1}
I20260924 04:43:52.204774 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.188173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.0.98.60:43085)
0924 04:43:52.188605 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.188624 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.188830 (+ 206us) server_negotiation.cc:407] Connection header received
0924 04:43:52.189546 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.189551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.189921 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.190282 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.191049 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.192168 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.193308 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.193820 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.193990 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.198697 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.198701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.198703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.198728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.201657 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.202168 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.202172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.202175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.202260 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.202887 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.202892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.202897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.203363 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.203521 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.204216 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.204328 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":4992,"thread_start_us":135,"threads_started":1}
I20260924 04:43:52.205883 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.195611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35350)
0924 04:43:52.195869 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.195883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:52.195975 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:52.196683 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:52.196687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:52.196877 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:52.196978 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.196985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.198282 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.198286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.199235 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.199248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.199918 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.199921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.200011 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.200433 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:52.200447 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:52.202071 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:52.204146 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.204152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.204156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.204414 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.204756 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.204760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.204763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.204862 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.205302 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:52.205305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:52.205408 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 04:43:52.205523 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:43:52.205677 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260924 04:43:52.206496 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.196118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.0.98.60:43085)
0924 04:43:52.196454 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.196464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.196475 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:43:52.196501 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.196503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.196540 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.197107 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.197197 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.198129 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.199401 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.199785 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.200218 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.202205 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.202212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.202217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.202241 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.204021 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.204543 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.204548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.204550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.204642 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.204985 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.204990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.204993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.205186 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.205282 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.206190 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.206305 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":8448,"thread_start_us":104,"threads_started":1}
I20260924 04:43:52.207857 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.194531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.98.62:39211)
0924 04:43:52.194670 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.194674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.195158 (+ 484us) server_negotiation.cc:407] Connection header received
0924 04:43:52.195478 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.195482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.195526 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.195574 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.196830 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.198004 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.199678 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.200041 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.200134 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.203522 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.203526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.203528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.203553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.205337 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.206328 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.206332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.206334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.206426 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.206761 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.206765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.206766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.207058 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.207165 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.207610 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.207722 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:43:52.208065 13801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: Bootstrap starting.
I20260924 04:43:52.208336 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.194475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39211 (local address 127.0.0.1:38750)
0924 04:43:52.194996 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.195058 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:52.195258 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:52.195744 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:52.195762 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:52.196497 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:52.196689 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.196698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.198398 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.198401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.199535 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.199550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.200326 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.200331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.200484 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.201605 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:52.201621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:52.203373 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:52.205598 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.205607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.205613 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.205878 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.206552 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.206558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.206562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.206649 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.207237 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:52.207243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:52.207372 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 04:43:52.207979 (+ 607us) client_negotiation.cc:240] Negotiation successful
0924 04:43:52.208158 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"spinlock_wait_cycles":14080,"thread_start_us":109,"threads_started":1}
I20260924 04:43:52.209517 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: Bootstrap starting.
I20260924 04:43:52.209558 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:52.210768 13801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:52.211673 13801 log.cc:824] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:52.212241 13876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:52.213174 13876 log.cc:824] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:52.214072 13801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: No bootstrap required, opened a new log
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:52.217314 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: No bootstrap required, opened a new log
I20260924 04:43:52.218731 13801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.218992 13801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:52.219066 13801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3456e280e95f4b92a41ba281f6cdf536, State: Initialized, Role: FOLLOWER
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:43:52.219797 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.208892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.0.98.62:39211)
0924 04:43:52.208991 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.208995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.209007 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:43:52.209040 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.209043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.209085 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.209159 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.210049 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.210828 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.211948 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.212317 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.212414 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.215809 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.215812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.215817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.215861 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.217531 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.218124 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.218128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.218129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.218219 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.218544 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.218548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.218549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.218783 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.218964 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.219443 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.219580 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:43:52.219666 13801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.222138 13801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:52.222676 13876 raft_consensus.cc:348] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.222997 13876 raft_consensus.cc:374] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:52.223099 13876 raft_consensus.cc:729] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f038525da28a4b3186d184065d0604ae, State: Initialized, Role: FOLLOWER
I20260924 04:43:52.223858 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.213425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.98.61:46629)
0924 04:43:52.213542 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.213546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.213562 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:43:52.213670 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.213673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.213723 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.213817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.214473 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.215269 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.216972 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.217330 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.217412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.219944 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.219949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.219952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.219985 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.221737 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.222262 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.222267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.222271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.222357 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.222679 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.222683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.222685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.222970 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.223053 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.223487 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.223635 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":48}
I20260924 04:43:52.224141 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.213176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46629 (local address 127.0.0.1:41588)
0924 04:43:52.213452 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.213471 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:52.213573 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:52.213942 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:52.213945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:52.214181 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:52.214323 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.214331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.215403 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.215406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.216576 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.216587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.217575 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.217578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.217674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.218162 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:52.218177 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:52.219770 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:52.221894 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.221897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.221899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.222139 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.222473 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.222476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.222478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.222575 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.223158 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:52.223161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:52.223283 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 04:43:52.223744 (+ 461us) client_negotiation.cc:240] Negotiation successful
0924 04:43:52.223918 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260924 04:43:52.223713 13876 consensus_queue.cc:260] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.225102 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:43:52.226166 13876 sys_catalog.cc:564] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:43:52.225744 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.229130 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 04:43:52 dist-test-slave-npg2 krb5kdc[13720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:43:52.228056 13733 master_runner.cc:186] Error getting master registration for 127.0.98.61:46629: OK, instance_id { permanent_uuid: "f038525da28a4b3186d184065d0604ae" instance_seqno: 1790225031939396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:43:52.232391 13733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 04:43:52.233629 13985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:43:52.233790 13985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 04:43:52.235651 13972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:43:52.235736 13972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:43:52.235915 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.225947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.0.98.60:43085)
0924 04:43:52.226081 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.226085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.226364 (+ 279us) server_negotiation.cc:407] Connection header received
0924 04:43:52.226505 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.226507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.226542 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.226586 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.227439 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.228152 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.229474 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.230060 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.230135 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.232468 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.232472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.232474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.232498 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.234153 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.234627 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.234630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.234632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.234703 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.234983 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.234986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.234987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.235224 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.235305 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.235601 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.235722 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:43:52.235934 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.236050 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:43:52.236361 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.226021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43085 (local address 127.0.0.1:35352)
0924 04:43:52.226304 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:52.226319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:52.226419 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:52.226711 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:52.226715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:52.226916 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:52.227022 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.227029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.228266 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.228269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.229356 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:52.229370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.230263 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.230266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:52.230348 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.230853 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:52.230869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:52.232330 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:52.234271 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.234274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.234277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.234527 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.234806 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:52.234809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:52.234811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:52.234887 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:52.235362 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:52.235365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:52.235449 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:43:52.236096 (+ 647us) client_negotiation.cc:240] Negotiation successful
0924 04:43:52.236231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260924 04:43:52.238940 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: Bootstrap starting.
I20260924 04:43:52.235594 13808 master_runner.cc:186] Error getting master registration for 127.0.98.60:43085: OK, instance_id { permanent_uuid: "46adede31efb4289a14ea5443475b66f" instance_seqno: 1790225032181172 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:43:52.239148 13808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:43:52.241336 13952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:52.242039 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.221310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.98.61:46629)
0924 04:43:52.221434 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.221438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.221596 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:43:52.221782 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.221785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.221831 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.221892 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.222728 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.223533 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.224686 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.225037 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.225111 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.228678 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.228682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.228684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.228713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.231191 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.239806 (+ 8615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.239811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.239812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.239936 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.240368 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.240374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.240376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.240664 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.240881 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.241747 (+ 866us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.241882 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260924 04:43:52.242363 13952 log.cc:824] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:52.244697 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: No bootstrap required, opened a new log
I20260924 04:43:52.245610 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:43:52.249936 13952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.250216 13952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:52.250278 13952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46adede31efb4289a14ea5443475b66f, State: Initialized, Role: FOLLOWER
I20260924 04:43:52.250818 13952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.251368 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.251561 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:43:52.252254 13952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:52.260586 13883 master_runner.cc:263] No leader master found from master 46adede31efb4289a14ea5443475b66f
I20260924 04:43:52.260663 13883 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 04:43:52.262399 14007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:43:52.262558 14007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:43:52.266233 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.248825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.0.98.60:43085)
0924 04:43:52.248940 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.248944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.256839 (+ 7895us) server_negotiation.cc:407] Connection header received
0924 04:43:52.256919 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.256922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.256969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.257045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.257944 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.258725 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.260479 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.260824 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.260895 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.262685 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.262689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.262692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.262718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.264390 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.264889 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.264893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.264894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.264978 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.265263 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.265266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.265268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.265474 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.265591 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.265951 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.266071 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:43:52.272353 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.248059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.0.98.62:39211)
0924 04:43:52.248191 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.248196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.254944 (+ 6748us) server_negotiation.cc:407] Connection header received
0924 04:43:52.255174 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.255178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.255234 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.255293 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.256968 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.257709 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.258856 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.259203 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.259282 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.268288 (+ 9006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.268294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.268297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.268330 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.270106 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.270624 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.270628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.270630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.270716 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.271011 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.271014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.271016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.271261 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.271397 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.272051 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.272176 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:43:52.277083 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:52.249179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.0.98.61:46629)
0924 04:43:52.249284 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:52.249288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:52.267858 (+ 18570us) server_negotiation.cc:407] Connection header received
0924 04:43:52.268031 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:52.268034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:52.268085 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:52.268134 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:52.269439 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.270237 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.271379 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:52.271724 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:52.271807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:52.273424 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:52.273428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:52.273431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:52.273458 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:52.275154 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.275706 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.275710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.275712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.275806 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:52.276097 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:52.276100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:52.276102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:52.276307 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:52.276421 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:52.276768 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 04:43:52.276916 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:43:52.427728 13905 tablet.cc:2369] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:43:52.435751 13829 tablet.cc:2369] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:43:52.442112 13996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:43:52.442329 13996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.443451 13996 leader_election.cc:288] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3456e280e95f4b92a41ba281f6cdf536 (127.0.98.62:39211), f038525da28a4b3186d184065d0604ae (127.0.98.61:46629)
I20260924 04:43:52.444115 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46adede31efb4289a14ea5443475b66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3456e280e95f4b92a41ba281f6cdf536" is_pre_election: true
I20260924 04:43:52.444115 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46adede31efb4289a14ea5443475b66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f038525da28a4b3186d184065d0604ae" is_pre_election: true
I20260924 04:43:52.444437 13850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46adede31efb4289a14ea5443475b66f in term 0.
I20260924 04:43:52.444437 13775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46adede31efb4289a14ea5443475b66f in term 0.
I20260924 04:43:52.444892 13900 leader_election.cc:302] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46adede31efb4289a14ea5443475b66f, f038525da28a4b3186d184065d0604ae; no voters:
I20260924 04:43:52.445222 13996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:43:52.445303 13996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:43:52.445351 13996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:52.446683 13996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.447098 13996 leader_election.cc:288] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [CANDIDATE]: Term 1 election: Requested vote from peers 3456e280e95f4b92a41ba281f6cdf536 (127.0.98.62:39211), f038525da28a4b3186d184065d0604ae (127.0.98.61:46629)
I20260924 04:43:52.447538 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46adede31efb4289a14ea5443475b66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3456e280e95f4b92a41ba281f6cdf536"
I20260924 04:43:52.447556 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46adede31efb4289a14ea5443475b66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f038525da28a4b3186d184065d0604ae"
I20260924 04:43:52.447677 13850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:52.447680 13775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:52.448911 13775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46adede31efb4289a14ea5443475b66f in term 1.
I20260924 04:43:52.448911 13850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46adede31efb4289a14ea5443475b66f in term 1.
I20260924 04:43:52.449222 13903 leader_election.cc:302] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3456e280e95f4b92a41ba281f6cdf536, 46adede31efb4289a14ea5443475b66f; no voters:
I20260924 04:43:52.449447 13996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:52.449699 13996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [term 1 LEADER]: Becoming Leader. State: Replica: 46adede31efb4289a14ea5443475b66f, State: Running, Role: LEADER
I20260924 04:43:52.450047 13996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } }
I20260924 04:43:52.452219 14009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46adede31efb4289a14ea5443475b66f. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.452375 14009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:52.452944 14011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:52.454074 14011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:52.460788 13850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [term 1 FOLLOWER]: Refusing update from remote peer 46adede31efb4289a14ea5443475b66f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:52.460820 13775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [term 1 FOLLOWER]: Refusing update from remote peer 46adede31efb4289a14ea5443475b66f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:52.461520 13996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:52.461732 14009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:43:52.462396 13754 tablet.cc:2369] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:43:52.466147 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46adede31efb4289a14ea5443475b66f. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.466147 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46adede31efb4289a14ea5443475b66f. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.466276 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:43:52.466276 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:43:52.468799 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.468932 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:52.469190 14013 mvcc.cc:205] Tried to move back new op lower bound from 7332761732951814144 to 7332761732918415360. Current Snapshot: MvccSnapshot[applied={T|T < 7332761732951814144}]
I20260924 04:43:52.469312 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.469421 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:43:52.469797 14012 mvcc.cc:205] Tried to move back new op lower bound from 7332761732951814144 to 7332761732918415360. Current Snapshot: MvccSnapshot[applied={T|T < 7332761732951814144}]
I20260924 04:43:52.470506 14009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.470633 14009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:52.471396 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46adede31efb4289a14ea5443475b66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3456e280e95f4b92a41ba281f6cdf536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39211 } } peers { permanent_uuid: "f038525da28a4b3186d184065d0604ae" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46629 } } peers { permanent_uuid: "46adede31efb4289a14ea5443475b66f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43085 } } }
I20260924 04:43:52.471534 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:43:52.471590 14014 mvcc.cc:205] Tried to move back new op lower bound from 7332761732951814144 to 7332761732918415360. Current Snapshot: MvccSnapshot[applied={T|T < 7332761732951814144}]
I20260924 04:43:52.472159 14011 catalog_manager.cc:1347] Generated new cluster ID: 0ca5f9ba0e34484e8fac61f4835ee296
I20260924 04:43:52.472224 14011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:43:52.485092 14011 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:43:52.486470 14011 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:43:52.493723 14011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46adede31efb4289a14ea5443475b66f: Generated new TSK 0
I20260924 04:43:52.494525 14011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:43:53.235297 13985 catalog_manager.cc:1259] Loaded cluster ID: 0ca5f9ba0e34484e8fac61f4835ee296
I20260924 04:43:53.235443 13985 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: loading cluster ID for follower catalog manager: success
I20260924 04:43:53.236804 13972 catalog_manager.cc:1259] Loaded cluster ID: 0ca5f9ba0e34484e8fac61f4835ee296
I20260924 04:43:53.236868 13972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: loading cluster ID for follower catalog manager: success
I20260924 04:43:53.237342 13985 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: acquiring CA information for follower catalog manager: success
I20260924 04:43:53.238495 13985 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f038525da28a4b3186d184065d0604ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:43:53.238529 13972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: acquiring CA information for follower catalog manager: success
I20260924 04:43:53.239589 13972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3456e280e95f4b92a41ba281f6cdf536: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:43:53.552506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13733
I20260924 04:43:53.559494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13808
I20260924 04:43:53.565382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 13883
2026-09-24T04:43:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3234 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:43:53 dist-test-slave-npg2 krb5kdc[14026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:43:53 dist-test-slave-npg2 krb5kdc[14026](info): set up 2 sockets
Sep 24 04:43:53 dist-test-slave-npg2 krb5kdc[14026](info): commencing operation
krb5kdc: starting...
W20260924 04:43:54.715631 392 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:43:54 dist-test-slave-npg2 krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:43:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:43:54.843086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:54.956729 14039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:54.957016 14039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:54.957058 14039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:54.961215 14039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:54.961275 14039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:54.961295 14039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:54.961311 14039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:54.961328 14039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:54.966646 14039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:54.967660 14039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:54.968684 14039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:54.977870 14045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:54.977856 14044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:54.978152 14039 server_base.cc:1034] running on GCE node
W20260924 04:43:54.977856 14047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:54.978869 14039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:54.980072 14039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:54.981297 14039 hybrid_clock.cc:648] HybridClock initialized: now 1790225034981266 us; error 49 us; skew 500 ppm
Sep 24 04:43:54 dist-test-slave-npg2 krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:54.984230 14039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:54.985160 14039 webserver.cc:460] Webserver started at http://127.0.98.62:41851/ using document root <none> and password file <none>
I20260924 04:43:54.985787 14039 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:54.985895 14039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:54.986150 14039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:54.988417 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "03877327709747e4a49609d0bf4c3e4f"
format_stamp: "Formatted at 2026-09-24 04:43:54 on dist-test-slave-npg2"
server_key: "00217e5333bbe6208760a3f1f4a5b3cb"
server_key_iv: "f1622a2678c1d74c436d657f0cb8ff58"
server_key_version: "encryptionkey@0"
I20260924 04:43:54.989077 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "03877327709747e4a49609d0bf4c3e4f"
format_stamp: "Formatted at 2026-09-24 04:43:54 on dist-test-slave-npg2"
server_key: "00217e5333bbe6208760a3f1f4a5b3cb"
server_key_iv: "f1622a2678c1d74c436d657f0cb8ff58"
server_key_version: "encryptionkey@0"
I20260924 04:43:54.993108 14039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:43:54.995879 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:54.997299 14039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:54.997457 14039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "03877327709747e4a49609d0bf4c3e4f"
format_stamp: "Formatted at 2026-09-24 04:43:54 on dist-test-slave-npg2"
server_key: "00217e5333bbe6208760a3f1f4a5b3cb"
server_key_iv: "f1622a2678c1d74c436d657f0cb8ff58"
server_key_version: "encryptionkey@0"
I20260924 04:43:54.997581 14039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:55.018597 14039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:55.022199 14039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:55.022431 14039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:55.029940 14039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41905
I20260924 04:43:55.029979 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41905 every 8 connection(s)
I20260924 04:43:55.030957 14039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:55.034003 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:55.039683 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f: Bootstrap starting.
I20260924 04:43:55.040038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14039
I20260924 04:43:55.040176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:43:55.040472 392 external_mini_cluster.cc:1428] Setting key 2a0b54791991cc0aad4a89dbde8f99e1
I20260924 04:43:55.042538 14107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:55.043464 14107 log.cc:824] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:55.045377 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f: No bootstrap required, opened a new log
I20260924 04:43:55.048977 14107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03877327709747e4a49609d0bf4c3e4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 04:43:55.049201 14107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:55.049263 14107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03877327709747e4a49609d0bf4c3e4f, State: Initialized, Role: FOLLOWER
Sep 24 04:43:55 dist-test-slave-npg2 krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:55.049769 14107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03877327709747e4a49609d0bf4c3e4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 04:43:55.049898 14107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:43:55.049989 14107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:43:55.050107 14107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:55.051299 14107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03877327709747e4a49609d0bf4c3e4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 04:43:55.051684 14107 leader_election.cc:302] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03877327709747e4a49609d0bf4c3e4f; no voters:
I20260924 04:43:55.052007 14107 leader_election.cc:288] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:43:55.052134 14112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:55.052384 14112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [term 1 LEADER]: Becoming Leader. State: Replica: 03877327709747e4a49609d0bf4c3e4f, State: Running, Role: LEADER
I20260924 04:43:55.052781 14112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03877327709747e4a49609d0bf4c3e4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 04:43:55.053388 14107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:55.054574 14114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03877327709747e4a49609d0bf4c3e4f. Latest consensus state: current_term: 1 leader_uuid: "03877327709747e4a49609d0bf4c3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03877327709747e4a49609d0bf4c3e4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } } }
I20260924 04:43:55.054715 14114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:55.055109 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03877327709747e4a49609d0bf4c3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03877327709747e4a49609d0bf4c3e4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } } }
I20260924 04:43:55.055305 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:55.056530 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:55.041875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.0.98.62:41905)
0924 04:43:55.042312 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:55.042328 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:55.042366 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:43:55.043089 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:55.043109 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:55.043640 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:55.043992 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:55.044815 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:55.045859 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:55.046723 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:55.047289 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:55.047461 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:55.050282 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:55.050287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:55.050301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:55.050328 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:55.052400 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:55.052976 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:55.052981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:55.052988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:55.053126 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:55.053459 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:55.053466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:55.053470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:55.053985 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:55.054206 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:55.055184 (+ 978us) server_negotiation.cc:299] Negotiation successful
0924 04:43:55.055519 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":141,"threads_started":1}
W20260924 04:43:55.058871 14127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:43:55.058949 14127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:43:55.059006 14121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:55.059720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:43:55.059878 14121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:55.065569 14121 catalog_manager.cc:1347] Generated new cluster ID: 9aa7592f345440cfb7e34fbbd93e4ea5
I20260924 04:43:55.065711 14121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:43:55.078639 14121 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:43:55.079674 14121 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:43:55.084364 14121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03877327709747e4a49609d0bf4c3e4f: Generated new TSK 0
I20260924 04:43:55.085054 14121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:43:55.116003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14039
2026-09-24T04:43:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1548 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:43:55 dist-test-slave-npg2 krb5kdc[14134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:43:55 dist-test-slave-npg2 krb5kdc[14134](info): set up 2 sockets
Sep 24 04:43:55 dist-test-slave-npg2 krb5kdc[14134](info): commencing operation
krb5kdc: starting...
W20260924 04:43:56.171407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:43:56 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:43:56.307421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:56.420679 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:56.421067 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:56.421173 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:56.425752 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:56.425884 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:56.425926 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:56.425967 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:56.426004 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:56.431689 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:56.432914 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:56.434026 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:56.443341 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:56.443384 14147 server_base.cc:1034] running on GCE node
W20260924 04:43:56.443401 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:56.443480 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:56.444211 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:56.445384 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:56.446568 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790225036446539 us; error 45 us; skew 500 ppm
Sep 24 04:43:56 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:56.449517 14147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:56.450415 14147 webserver.cc:460] Webserver started at http://127.0.98.62:35915/ using document root <none> and password file <none>
I20260924 04:43:56.450956 14147 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:56.451014 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:56.451262 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:56.453332 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "31e5c0e1aebe4647ab496aad06aabf04"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "d0ff7be98f02b38df4ed5c07b487a2b9"
server_key_iv: "dd9d270db9f6ec2931ebe2887f7e6adf"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.453861 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "31e5c0e1aebe4647ab496aad06aabf04"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "d0ff7be98f02b38df4ed5c07b487a2b9"
server_key_iv: "dd9d270db9f6ec2931ebe2887f7e6adf"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.457702 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:43:56.460325 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:56.461644 14147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:56.461751 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "31e5c0e1aebe4647ab496aad06aabf04"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "d0ff7be98f02b38df4ed5c07b487a2b9"
server_key_iv: "dd9d270db9f6ec2931ebe2887f7e6adf"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.461877 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:56.477844 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:56.481333 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:56.481503 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:56.488896 14147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40439
I20260924 04:43:56.489225 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40439 every 8 connection(s)
I20260924 04:43:56.490031 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:56.493340 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:56.494356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14147
I20260924 04:43:56.494508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:43:56.494863 392 external_mini_cluster.cc:1428] Setting key fad551c3a52899a7dec7762d9ead8893
I20260924 04:43:56.500624 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap starting.
I20260924 04:43:56.503631 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:56.504642 14215 log.cc:824] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:43:56 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:56.506772 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: No bootstrap required, opened a new log
I20260924 04:43:56.510541 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:56.496391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.0.98.62:40439)
0924 04:43:56.496828 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:56.496842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:56.496879 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:43:56.497587 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:56.497608 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:56.497968 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:56.498277 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:56.499351 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:56.500366 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:56.501633 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:56.502247 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:56.502382 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:56.505660 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:56.505665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:56.505680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:56.505742 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:56.507627 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:56.508105 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:56.508112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:56.508116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:56.508248 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:56.508543 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:56.508547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:56.508549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:56.508960 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:56.509168 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:56.509478 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:43:56.509784 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
I20260924 04:43:56.510854 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:56.511098 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:56.511202 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e5c0e1aebe4647ab496aad06aabf04, State: Initialized, Role: FOLLOWER
I20260924 04:43:56.511703 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:56.511880 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:43:56.511941 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:43:56.512043 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:56.513484 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:56.513818 14215 leader_election.cc:302] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e5c0e1aebe4647ab496aad06aabf04; no voters:
I20260924 04:43:56.514135 14215 leader_election.cc:288] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:43:56.514367 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:56.514628 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 LEADER]: Becoming Leader. State: Replica: 31e5c0e1aebe4647ab496aad06aabf04, State: Running, Role: LEADER
I20260924 04:43:56.514967 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:56.515561 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:56.516692 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e5c0e1aebe4647ab496aad06aabf04. Latest consensus state: current_term: 1 leader_uuid: "31e5c0e1aebe4647ab496aad06aabf04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260924 04:43:56.516819 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:56.517184 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:56.518122 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e5c0e1aebe4647ab496aad06aabf04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260924 04:43:56.518265 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:56.520272 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:56.525846 14229 catalog_manager.cc:1347] Generated new cluster ID: 669dbe67049246158c7f5dce652b6642
I20260924 04:43:56.525930 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:43:56.532745 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:43:56.533725 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:43:56.541584 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Generated new TSK 0
I20260924 04:43:56.542392 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:43:56.583307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:56.702925 14241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:56.703222 14241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:56.703265 14241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:56.707468 14241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:56.707540 14241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:56.707633 14241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:43:56.713016 14241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:56.714203 14241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:56.715329 14241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:56.725483 14247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:56.725600 14249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:56.725541 14246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:56.725536 14241 server_base.cc:1034] running on GCE node
I20260924 04:43:56.726152 14241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:56.726789 14241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:56.727958 14241 hybrid_clock.cc:648] HybridClock initialized: now 1790225036727922 us; error 49 us; skew 500 ppm
Sep 24 04:43:56 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:56.735962 14241 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:43:56.737236 14241 webserver.cc:460] Webserver started at http://127.0.98.1:36161/ using document root <none> and password file <none>
I20260924 04:43:56.737845 14241 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:56.737913 14241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:56.738101 14241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:56.740447 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1d09b146dd642f191903131c90cecd2"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "94be01fc365327147495e1a9b4f1111d"
server_key_iv: "82aad18e6974218427a18d19481fbad4"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.741027 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1d09b146dd642f191903131c90cecd2"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "94be01fc365327147495e1a9b4f1111d"
server_key_iv: "82aad18e6974218427a18d19481fbad4"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.745260 14241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:43:56.747931 14256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:56.749317 14241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:56.749466 14241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "c1d09b146dd642f191903131c90cecd2"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "94be01fc365327147495e1a9b4f1111d"
server_key_iv: "82aad18e6974218427a18d19481fbad4"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.749604 14241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:56.763185 14241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:56.766765 14241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:56.766973 14241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:56.767619 14241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:56.768451 14241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:56.768522 14241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:56.768587 14241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:56.768639 14241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:56.778321 14241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38931
I20260924 04:43:56.778338 14369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38931 every 8 connection(s)
I20260924 04:43:56.779462 14241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:43:56.780843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14241
I20260924 04:43:56.780988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:43:56.781282 392 external_mini_cluster.cc:1428] Setting key be942bd61c790d3e5ebfcb839edb3b37
Sep 24 04:43:56 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:43:56.795771 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:56.781642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42135 (local address 127.0.98.62:40439)
0924 04:43:56.781793 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:56.781797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:56.782951 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 04:43:56.784996 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:56.785001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:56.785060 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:56.785162 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:56.786348 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:56.787254 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:56.788456 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:56.788918 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:56.789060 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:56.792028 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:56.792031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:56.792034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:56.792058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:56.793449 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:56.793975 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:56.793977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:56.793979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:56.794102 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:56.794517 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:56.794520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:56.794521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:56.794704 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:56.794835 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:56.795523 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 04:43:56.795638 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:43:56.797173 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:56.782133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:42135)
0924 04:43:56.782787 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:56.782842 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:56.784729 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:56.785322 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:56.785328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:56.785713 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:56.786186 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:56.786199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:56.787425 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:56.787428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:56.788321 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:56.788330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:56.789081 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:56.789085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:56.789272 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:56.789927 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:56.789944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:56.791845 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:56.793576 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:56.793581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:56.793594 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:56.793859 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:56.794220 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:56.794226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:56.794228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:56.794383 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:56.794843 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:56.794854 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:56.795244 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 04:43:56.795476 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 04:43:56.795881 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":154,"threads_started":1}
I20260924 04:43:56.798534 14370 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260924 04:43:56.798807 14370 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:56.799407 14370 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260924 04:43:56.800889 14179 ts_manager.cc:194] Registered new tserver with Master: c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931)
I20260924 04:43:56.802016 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:43:56.810357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:56.926658 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:56.926956 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:56.927001 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:56.931114 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:56.931224 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:56.931384 14375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:43:56.937084 14375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:56.938390 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:56.939615 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:56.949643 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:56.949788 14375 server_base.cc:1034] running on GCE node
W20260924 04:43:56.949685 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:56.949667 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:56.950443 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:56.951233 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:56.952435 14375 hybrid_clock.cc:648] HybridClock initialized: now 1790225036952420 us; error 36 us; skew 500 ppm
Sep 24 04:43:56 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:56.955878 14375 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:43:56.956804 14375 webserver.cc:460] Webserver started at http://127.0.98.2:34679/ using document root <none> and password file <none>
I20260924 04:43:56.957355 14375 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:56.957417 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:56.957597 14375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:56.959756 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "d82fb1b585434f0dadb086f4b2c8817e"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "e4e7a605794f257682cb844161c25f54"
server_key_iv: "930ec05e28ddbad0a6dcaf5ea1c14787"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.960302 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d82fb1b585434f0dadb086f4b2c8817e"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "e4e7a605794f257682cb844161c25f54"
server_key_iv: "930ec05e28ddbad0a6dcaf5ea1c14787"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.964267 14375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:43:56.967079 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:56.968537 14375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:43:56.968680 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "d82fb1b585434f0dadb086f4b2c8817e"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "e4e7a605794f257682cb844161c25f54"
server_key_iv: "930ec05e28ddbad0a6dcaf5ea1c14787"
server_key_version: "encryptionkey@0"
I20260924 04:43:56.968807 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:57.014072 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:57.018182 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:57.018410 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:57.019089 14375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:57.020094 14375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:57.020144 14375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.020188 14375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:57.020206 14375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.030157 14375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32905
I20260924 04:43:57.030177 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32905 every 8 connection(s)
I20260924 04:43:57.031411 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:43:57.037914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14375
I20260924 04:43:57.038056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:43:57.038424 392 external_mini_cluster.cc:1428] Setting key cecd8c2f53650f5ca8e1ae6b4be8757e
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:57.049366 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.033387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49901 (local address 127.0.98.62:40439)
0924 04:43:57.033576 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.033581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.034576 (+ 995us) server_negotiation.cc:407] Connection header received
0924 04:43:57.035975 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.035978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.036028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.036233 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:57.037518 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.038224 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.039831 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.040135 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.040230 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.044331 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:57.044334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:57.044336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:57.044365 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:57.046509 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.047338 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.047343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.047345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.047457 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.047922 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.047926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.047928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.048217 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:57.048334 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.048960 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.049129 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:43:57.050187 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.033868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:49901)
0924 04:43:57.034400 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.034454 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.035740 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.036410 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.036418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.036844 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:57.037345 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.037359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.038388 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.038392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.039674 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.039688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.040453 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.040459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.040718 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.041707 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:57.041734 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:57.044007 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:57.046701 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.046707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.046724 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.047069 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.047570 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.047572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.047574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.047788 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.048338 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:57.048344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:57.048629 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.048838 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.049132 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":156,"threads_started":1}
I20260924 04:43:57.051788 14504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260924 04:43:57.052090 14504 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:57.052676 14504 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:43:57.053951 14179 ts_manager.cc:194] Registered new tserver with Master: d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905)
I20260924 04:43:57.054924 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:43:57.070724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:57.185678 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:57.185973 14509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:57.186053 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:57.190284 14509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:57.190383 14509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:57.190485 14509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:43:57.195940 14509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:57.197029 14509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:57.198051 14509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:57.207919 14517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:57.207919 14515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:57.207944 14514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:57.208247 14509 server_base.cc:1034] running on GCE node
I20260924 04:43:57.208776 14509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:57.209374 14509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:57.210597 14509 hybrid_clock.cc:648] HybridClock initialized: now 1790225037210568 us; error 39 us; skew 500 ppm
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:57.213675 14509 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:43:57.214654 14509 webserver.cc:460] Webserver started at http://127.0.98.3:46275/ using document root <none> and password file <none>
I20260924 04:43:57.215317 14509 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:57.215410 14509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:57.215665 14509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:43:57.217988 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "4c947446296a4657a93d8d82e73db77b"
format_stamp: "Formatted at 2026-09-24 04:43:57 on dist-test-slave-npg2"
server_key: "6a1af69af4ae73711b18226a8669b6ee"
server_key_iv: "fb2bd3a6fe8afbd32b5815acdeaa5d2c"
server_key_version: "encryptionkey@0"
I20260924 04:43:57.218565 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4c947446296a4657a93d8d82e73db77b"
format_stamp: "Formatted at 2026-09-24 04:43:57 on dist-test-slave-npg2"
server_key: "6a1af69af4ae73711b18226a8669b6ee"
server_key_iv: "fb2bd3a6fe8afbd32b5815acdeaa5d2c"
server_key_version: "encryptionkey@0"
I20260924 04:43:57.222563 14509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:43:57.225270 14524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.226609 14509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:43:57.226773 14509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "4c947446296a4657a93d8d82e73db77b"
format_stamp: "Formatted at 2026-09-24 04:43:57 on dist-test-slave-npg2"
server_key: "6a1af69af4ae73711b18226a8669b6ee"
server_key_iv: "fb2bd3a6fe8afbd32b5815acdeaa5d2c"
server_key_version: "encryptionkey@0"
I20260924 04:43:57.226897 14509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:57.239320 14509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:57.242904 14509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:57.243175 14509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:57.243806 14509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:57.244676 14509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:57.244746 14509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.244818 14509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:57.244867 14509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.254763 14509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38529
I20260924 04:43:57.254782 14637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38529 every 8 connection(s)
I20260924 04:43:57.255749 14509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:43:57.257954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14509
I20260924 04:43:57.258067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:43:57.258373 392 external_mini_cluster.cc:1428] Setting key 4030dcb0de84595b31320840ac439cc4
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:57.270438 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.257805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47107 (local address 127.0.98.62:40439)
0924 04:43:57.257946 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.257950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.258868 (+ 918us) server_negotiation.cc:407] Connection header received
0924 04:43:57.260332 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.260335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.260391 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.260483 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:57.261804 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.262343 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.263662 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.263885 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.263974 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.266881 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:57.266885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:57.266887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:57.266918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:57.268560 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.269047 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.269050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.269051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.269124 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.269458 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.269460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.269462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.269636 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:57.269714 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.270208 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.270310 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:43:57.271518 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.258245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:47107)
0924 04:43:57.258710 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.258766 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.260090 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.260776 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.260782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.261133 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:57.261623 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.261659 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.262463 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.262465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.263514 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.263523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.264031 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.264034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.264190 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.264887 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:57.264904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:57.266686 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:57.268685 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.268690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.268705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.268959 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.269243 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.269245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.269246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.269379 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.269724 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:57.269729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:57.269967 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.270125 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.270394 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":148,"threads_started":1}
I20260924 04:43:57.272959 14638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260924 04:43:57.273259 14638 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:57.273788 14638 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260924 04:43:57.274816 14178 ts_manager.cc:194] Registered new tserver with Master: 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
I20260924 04:43:57.275470 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47107
I20260924 04:43:57.284718 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:43:57.295300 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.286897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.98.62:40439)
0924 04:43:57.287061 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.287066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.287318 (+ 252us) server_negotiation.cc:407] Connection header received
0924 04:43:57.287474 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.287477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.287527 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.287614 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:57.288324 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.288856 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.289725 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.289948 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.290058 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.291647 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:57.291651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:57.291653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:57.291679 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:57.293230 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.293747 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.293758 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.293759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.293840 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.294251 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.294254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.294256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.294504 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:57.294707 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.294952 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.295090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:43:57.298890 14178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:43:57.302134 14178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:43:57.318565 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.312255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38529 (local address 127.0.0.1:57366)
0924 04:43:57.313125 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.313182 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.313311 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.313978 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.313984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.314005 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:57.314185 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.314196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.315692 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.315696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.316875 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.316901 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.318003 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.318007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.318125 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.318162 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.318276 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.318400 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":19,"thread_start_us":81,"threads_started":1}
I20260924 04:43:57.318681 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.312588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.3:38529)
0924 04:43:57.313084 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.313102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.313461 (+ 359us) server_negotiation.cc:407] Connection header received
0924 04:43:57.313587 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.313593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.313811 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.313991 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:57.314319 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.315551 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.316997 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.317659 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.318007 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.318171 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.318385 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.318504 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":140,"threads_started":1}
I20260924 04:43:57.319674 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.312265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38931 (local address 127.0.0.1:56554)
0924 04:43:57.313459 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.313472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.313546 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.314220 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.314225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.314247 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:57.314344 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.314352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.315455 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.315458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.317099 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.317107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.318959 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.318963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.319070 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.319096 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.319302 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.319452 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":331,"thread_start_us":100,"threads_started":1}
I20260924 04:43:57.320243 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.312972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.0.1:48728)
0924 04:43:57.313659 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.313673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.313768 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.314397 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.314401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.314423 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:57.314531 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.314537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.316156 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.316159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.317983 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.317996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.319766 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.319770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.319904 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.319939 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.320022 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.320105 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260924 04:43:57.320487 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.312355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.98.1:38931)
0924 04:43:57.313793 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.313798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.313817 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:43:57.313880 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.313883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.314085 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.314234 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:57.314441 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.315328 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.317220 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.318552 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.318722 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.318885 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.320223 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.320343 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":89,"threads_started":1}
I20260924 04:43:57.320823 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.313333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.0.98.2:32905)
0924 04:43:57.313653 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.313667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.313764 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:43:57.313877 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.313882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.314093 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.314234 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:57.314669 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.316009 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.318323 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.319428 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.319548 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.319694 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.320563 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.320662 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260924 04:43:57.322701 14572 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa7c5640fae4902be2113d64f8099fe (DEFAULT_TABLE table=test-table [id=18965779860d4d609070ec98648a0b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:57.322956 14304 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa7c5640fae4902be2113d64f8099fe (DEFAULT_TABLE table=test-table [id=18965779860d4d609070ec98648a0b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:57.323524 14438 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa7c5640fae4902be2113d64f8099fe (DEFAULT_TABLE table=test-table [id=18965779860d4d609070ec98648a0b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:57.323992 14304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa7c5640fae4902be2113d64f8099fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:57.324004 14572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa7c5640fae4902be2113d64f8099fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:57.324548 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa7c5640fae4902be2113d64f8099fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:57.329820 14655 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Bootstrap starting.
I20260924 04:43:57.331532 14656 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Bootstrap starting.
I20260924 04:43:57.331938 14657 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Bootstrap starting.
I20260924 04:43:57.332932 14655 tablet_bootstrap.cc:654] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:57.333729 14655 log.cc:824] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:57.334221 14656 tablet_bootstrap.cc:654] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:57.335064 14657 tablet_bootstrap.cc:654] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:57.335553 14656 log.cc:824] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:57.335875 14655 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: No bootstrap required, opened a new log
I20260924 04:43:57.335875 14657 log.cc:824] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:57.336192 14655 ts_tablet_manager.cc:1397] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:43:57.337260 14656 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: No bootstrap required, opened a new log
I20260924 04:43:57.337452 14656 ts_tablet_manager.cc:1397] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:43:57.339299 14657 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: No bootstrap required, opened a new log
I20260924 04:43:57.339473 14657 ts_tablet_manager.cc:1397] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260924 04:43:57.340582 14655 raft_consensus.cc:348] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.340860 14655 raft_consensus.cc:374] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:57.340939 14655 raft_consensus.cc:729] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d09b146dd642f191903131c90cecd2, State: Initialized, Role: FOLLOWER
I20260924 04:43:57.341274 14656 raft_consensus.cc:348] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.341526 14656 raft_consensus.cc:374] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:57.341491 14655 consensus_queue.cc:260] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.341615 14656 raft_consensus.cc:729] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c947446296a4657a93d8d82e73db77b, State: Initialized, Role: FOLLOWER
I20260924 04:43:57.342053 14656 consensus_queue.cc:260] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.342666 14655 ts_tablet_manager.cc:1428] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:43:57.342811 14638 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260924 04:43:57.343037 14370 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260924 04:43:57.343247 14657 raft_consensus.cc:348] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.343498 14657 raft_consensus.cc:374] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:57.343600 14657 raft_consensus.cc:729] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82fb1b585434f0dadb086f4b2c8817e, State: Initialized, Role: FOLLOWER
I20260924 04:43:57.344002 14657 consensus_queue.cc:260] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.344235 14656 ts_tablet_manager.cc:1428] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:43:57.346442 14657 ts_tablet_manager.cc:1428] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:43:57.347066 14504 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260924 04:43:57.375435 14662 raft_consensus.cc:482] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:43:57.375734 14662 raft_consensus.cc:504] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.376981 14662 leader_election.cc:288] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931), d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905)
I20260924 04:43:57.380995 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.377446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60435 (local address 127.0.98.1:38931)
0924 04:43:57.377585 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.377590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.377608 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:43:57.377664 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.377667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.377721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.377800 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:57.378342 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.378858 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.379963 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.380425 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.380555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.380618 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.380804 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.380876 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:43:57.380995 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.377311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38931 (local address 127.0.98.3:60435)
0924 04:43:57.377450 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.377468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.377565 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.377922 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.377926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.377962 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:57.378197 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.378208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.378974 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.378980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.379828 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.379837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.380558 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.380562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.380667 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.380696 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.380785 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.380858 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":22}
I20260924 04:43:57.381373 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.377840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48389 (local address 127.0.98.2:32905)
0924 04:43:57.377962 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.377968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.378041 (+ 73us) server_negotiation.cc:407] Connection header received
0924 04:43:57.378263 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.378268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.378317 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.378408 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:57.378713 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.379274 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.380343 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.380769 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.380895 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.380952 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.381174 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.381262 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:43:57.381726 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.377586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.98.3:48389)
0924 04:43:57.377962 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.377975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.378104 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.378420 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.378422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.378438 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:57.378593 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.378600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.379382 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.379387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.380213 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.380222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.380892 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.380895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.380994 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.381015 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.381531 (+ 516us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.381610 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":18,"thread_start_us":132,"threads_started":1}
I20260924 04:43:57.381698 14324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa7c5640fae4902be2113d64f8099fe" candidate_uuid: "4c947446296a4657a93d8d82e73db77b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d09b146dd642f191903131c90cecd2" is_pre_election: true
I20260924 04:43:57.382053 14324 raft_consensus.cc:2454] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c947446296a4657a93d8d82e73db77b in term 0.
I20260924 04:43:57.382404 14458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa7c5640fae4902be2113d64f8099fe" candidate_uuid: "4c947446296a4657a93d8d82e73db77b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82fb1b585434f0dadb086f4b2c8817e" is_pre_election: true
I20260924 04:43:57.382583 14528 leader_election.cc:302] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c947446296a4657a93d8d82e73db77b, c1d09b146dd642f191903131c90cecd2; no voters:
I20260924 04:43:57.382675 14458 raft_consensus.cc:2454] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c947446296a4657a93d8d82e73db77b in term 0.
I20260924 04:43:57.382870 14662 raft_consensus.cc:2781] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:43:57.382965 14662 raft_consensus.cc:482] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:43:57.383026 14662 raft_consensus.cc:3037] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:57.384150 14662 raft_consensus.cc:504] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.384486 14662 leader_election.cc:288] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [CANDIDATE]: Term 1 election: Requested vote from peers c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931), d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905)
I20260924 04:43:57.384852 14324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa7c5640fae4902be2113d64f8099fe" candidate_uuid: "4c947446296a4657a93d8d82e73db77b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d09b146dd642f191903131c90cecd2"
I20260924 04:43:57.384912 14458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa7c5640fae4902be2113d64f8099fe" candidate_uuid: "4c947446296a4657a93d8d82e73db77b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82fb1b585434f0dadb086f4b2c8817e"
I20260924 04:43:57.385001 14324 raft_consensus.cc:3037] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:57.385051 14458 raft_consensus.cc:3037] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:57.386114 14324 raft_consensus.cc:2454] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c947446296a4657a93d8d82e73db77b in term 1.
I20260924 04:43:57.386380 14458 raft_consensus.cc:2454] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c947446296a4657a93d8d82e73db77b in term 1.
I20260924 04:43:57.386407 14528 leader_election.cc:302] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c947446296a4657a93d8d82e73db77b, c1d09b146dd642f191903131c90cecd2; no voters:
I20260924 04:43:57.386598 14662 raft_consensus.cc:2781] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:57.386837 14662 raft_consensus.cc:686] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 1 LEADER]: Becoming Leader. State: Replica: 4c947446296a4657a93d8d82e73db77b, State: Running, Role: LEADER
I20260924 04:43:57.387243 14662 consensus_queue.cc:237] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.390734 14179 catalog_manager.cc:5495] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c947446296a4657a93d8d82e73db77b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4c947446296a4657a93d8d82e73db77b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } health_report { overall_health: UNKNOWN } } }
I20260924 04:43:57.401504 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.397932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.98.3:38529)
0924 04:43:57.398065 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.398070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.398087 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:43:57.398149 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.398151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.398203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.398297 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:43:57.398536 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.399026 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.400024 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.400292 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.400381 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.400694 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:43:57.401157 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:43:57.401252 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.401328 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.401374 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:43:57.407563 14324 raft_consensus.cc:1261] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Refusing update from remote peer 4c947446296a4657a93d8d82e73db77b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:57.407562 14458 raft_consensus.cc:1261] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Refusing update from remote peer 4c947446296a4657a93d8d82e73db77b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:57.408344 14662 consensus_queue.cc:1035] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:57.408634 14665 consensus_queue.cc:1035] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:57.417804 14668 mvcc.cc:205] Tried to move back new op lower bound from 7332761753213530112 to 7332761753141256192. Current Snapshot: MvccSnapshot[applied={T|T < 7332761753213530112}]
I20260924 04:43:57.427964 14179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:43:57.428189 14179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:43:57.431252 14179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Sending DeleteTablet for 3 replicas of tablet 0aa7c5640fae4902be2113d64f8099fe
I20260924 04:43:57.432215 14304 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa7c5640fae4902be2113d64f8099fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56554
I20260924 04:43:57.432210 14438 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa7c5640fae4902be2113d64f8099fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48728
I20260924 04:43:57.432258 14572 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa7c5640fae4902be2113d64f8099fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260924 04:43:57.432762 14677 tablet_replica.cc:330] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: stopping tablet replica
I20260924 04:43:57.432933 14679 tablet_replica.cc:330] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: stopping tablet replica
I20260924 04:43:57.432968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14241
I20260924 04:43:57.433156 14677 raft_consensus.cc:2229] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:57.433470 14677 raft_consensus.cc:2258] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:57.434280 14679 raft_consensus.cc:2229] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:43:57.434609 14679 raft_consensus.cc:2258] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:57.434974 14678 tablet_replica.cc:330] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: stopping tablet replica
I20260924 04:43:57.435276 14678 raft_consensus.cc:2229] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:57.435534 14678 raft_consensus.cc:2258] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:57.437166 14679 ts_tablet_manager.cc:1905] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:43:57.437166 14678 ts_tablet_manager.cc:1905] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:43:57.440163 14678 ts_tablet_manager.cc:1918] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:43:57.440217 14679 ts_tablet_manager.cc:1918] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:43:57.440312 14679 log.cc:1196] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/0aa7c5640fae4902be2113d64f8099fe
I20260924 04:43:57.440322 14678 log.cc:1196] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/0aa7c5640fae4902be2113d64f8099fe
I20260924 04:43:57.440714 14679 ts_tablet_manager.cc:1939] T 0aa7c5640fae4902be2113d64f8099fe P 4c947446296a4657a93d8d82e73db77b: Deleting consensus metadata
I20260924 04:43:57.440721 14678 ts_tablet_manager.cc:1939] T 0aa7c5640fae4902be2113d64f8099fe P d82fb1b585434f0dadb086f4b2c8817e: Deleting consensus metadata
I20260924 04:43:57.442440 14166 catalog_manager.cc:4841] TS d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): tablet 0aa7c5640fae4902be2113d64f8099fe (table test-table [id=18965779860d4d609070ec98648a0b93]) successfully deleted
I20260924 04:43:57.442703 14165 catalog_manager.cc:4841] TS 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): tablet 0aa7c5640fae4902be2113d64f8099fe (table test-table [id=18965779860d4d609070ec98648a0b93]) successfully deleted
W20260924 04:43:57.443419 14166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38931 (error 108)
I20260924 04:43:57.443542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14375
I20260924 04:43:57.444891 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.444597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38931 (local address 127.0.0.1:56564)
0924 04:43:57.444746 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.444818 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 04:43:57.445086 14166 catalog_manager.cc:4584] TS c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0aa7c5640fae4902be2113d64f8099fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38931: connect: Connection refused (error 111)
I20260924 04:43:57.452031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14509
I20260924 04:43:57.457113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14147
I20260924 04:43:57.463928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--webserver_interface=127.0.98.62
--webserver_port=35915
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:57.580265 14681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:57.580632 14681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:57.580739 14681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:57.585323 14681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:57.585418 14681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:57.585457 14681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:57.585513 14681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:57.585559 14681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:57.591562 14681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:57.592731 14681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:57.593839 14681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:57.607033 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:57.607082 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:57.607224 14681 server_base.cc:1034] running on GCE node
W20260924 04:43:57.607038 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:57.608042 14681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:57.609346 14681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:57.610555 14681 hybrid_clock.cc:648] HybridClock initialized: now 1790225037610537 us; error 34 us; skew 500 ppm
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:57.613531 14681 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:57.614575 14681 webserver.cc:460] Webserver started at http://127.0.98.62:35915/ using document root <none> and password file <none>
I20260924 04:43:57.615253 14681 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:57.615362 14681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:57.619463 14681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:57.621803 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.623212 14681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:57.623350 14681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "31e5c0e1aebe4647ab496aad06aabf04"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "d0ff7be98f02b38df4ed5c07b487a2b9"
server_key_iv: "dd9d270db9f6ec2931ebe2887f7e6adf"
server_key_version: "encryptionkey@0"
I20260924 04:43:57.623984 14681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:57.640628 14681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:57.643988 14681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:57.644214 14681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:57.651736 14681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40439
I20260924 04:43:57.652012 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40439 every 8 connection(s)
I20260924 04:43:57.652948 14681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:57.657378 14750 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:43:57.659159 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap starting.
I20260924 04:43:57.660887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14681
I20260924 04:43:57.661433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38931
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40439
--webserver_port=36161
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:57.676468 14750 log.cc:824] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:57.685549 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:43:57.685946 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap complete.
I20260924 04:43:57.690088 14750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:57.690505 14750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e5c0e1aebe4647ab496aad06aabf04, State: Initialized, Role: FOLLOWER
I20260924 04:43:57.690990 14750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:57.691135 14750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:43:57.691254 14750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:43:57.691376 14750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:43:57.693261 14750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:57.693717 14750 leader_election.cc:302] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e5c0e1aebe4647ab496aad06aabf04; no voters:
I20260924 04:43:57.694314 14750 leader_election.cc:288] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:43:57.694448 14754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:43:57.694694 14754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 LEADER]: Becoming Leader. State: Replica: 31e5c0e1aebe4647ab496aad06aabf04, State: Running, Role: LEADER
I20260924 04:43:57.695016 14754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:57.695659 14750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:57.696552 14755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31e5c0e1aebe4647ab496aad06aabf04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260924 04:43:57.696694 14755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:57.697067 14756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e5c0e1aebe4647ab496aad06aabf04. Latest consensus state: current_term: 2 leader_uuid: "31e5c0e1aebe4647ab496aad06aabf04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260924 04:43:57.697207 14756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:57.697517 14763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:57.702342 14763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:57.702951 14763 catalog_manager.cc:1259] Loaded cluster ID: 669dbe67049246158c7f5dce652b6642
I20260924 04:43:57.703012 14763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:43:57.704409 14763 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 04:43:57.709378 14763 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 04:43:57.715116 14763 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:43:57.716012 14763 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Loaded TSK: 0
I20260924 04:43:57.717208 14763 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:43:57.778240 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:57.778568 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:57.778669 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:57.782858 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:57.782969 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:57.783071 14752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:43:57.788905 14752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36161
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:57.790009 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:57.791101 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:57.802505 14777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:57.802510 14778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:57.802543 14780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:57.803210 14752 server_base.cc:1034] running on GCE node
I20260924 04:43:57.803663 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:57.804427 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:57.805680 14752 hybrid_clock.cc:648] HybridClock initialized: now 1790225037805666 us; error 31 us; skew 500 ppm
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:57.808911 14752 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:43:57.809928 14752 webserver.cc:460] Webserver started at http://127.0.98.1:36161/ using document root <none> and password file <none>
I20260924 04:43:57.810570 14752 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:57.810684 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:57.814955 14752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:57.817692 14787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:57.819298 14752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:57.819440 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "c1d09b146dd642f191903131c90cecd2"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "94be01fc365327147495e1a9b4f1111d"
server_key_iv: "82aad18e6974218427a18d19481fbad4"
server_key_version: "encryptionkey@0"
I20260924 04:43:57.820027 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:57.831575 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:57.834913 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:57.835173 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:57.835788 14752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:57.836906 14794 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:43:57.842532 14752 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:43:57.842641 14752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:43:57.842729 14752 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:43:57.844606 14752 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:43:57.844681 14752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:43:57.844947 14794 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Bootstrap starting.
I20260924 04:43:57.855505 14752 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38931
I20260924 04:43:57.855554 14901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38931 every 8 connection(s)
I20260924 04:43:57.856540 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:43:57.858183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14752
I20260924 04:43:57.858685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32905
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40439
--webserver_port=34679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:57.864872 14794 log.cc:824] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:57.873034 14794 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:43:57.873515 14794 tablet_bootstrap.cc:492] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Bootstrap complete.
I20260924 04:43:57.874933 14794 ts_tablet_manager.cc:1397] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260924 04:43:57.878221 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.858866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39759 (local address 127.0.98.62:40439)
0924 04:43:57.860708 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.860734 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.860793 (+ 59us) server_negotiation.cc:407] Connection header received
0924 04:43:57.862021 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.862033 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.862344 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.862634 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:57.864171 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.865969 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.867663 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.868503 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.868718 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.872011 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:57.872017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:57.872042 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:57.872080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:57.874392 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.875081 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.875088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.875093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.875302 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.875719 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.875723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.875724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.876207 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:57.876346 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.877058 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.877373 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1671,"thread_start_us":120,"threads_started":1}
I20260924 04:43:57.878057 14794 raft_consensus.cc:348] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.878258 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.858851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:39759)
0924 04:43:57.860188 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.860249 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.861551 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.862789 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.862799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.863322 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:57.863945 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.863966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.866185 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.866190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.867521 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.867532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.868667 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.868671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.868853 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.869606 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:57.869627 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:57.871756 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:57.874523 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.874530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.874545 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.874916 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.875407 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.875411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.875413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.875593 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.876330 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:57.876337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:57.876655 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.876850 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.877205 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"thread_start_us":193,"threads_started":1}
I20260924 04:43:57.878604 14794 raft_consensus.cc:729] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d09b146dd642f191903131c90cecd2, State: Initialized, Role: FOLLOWER
I20260924 04:43:57.879518 14794 consensus_queue.cc:260] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260924 04:43:57.880525 14794 ts_tablet_manager.cc:1428] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260924 04:43:57.881590 14902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260924 04:43:57.881832 14902 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:57.882391 14902 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260924 04:43:57.885325 14714 ts_manager.cc:194] Registered new tserver with Master: c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931)
I20260924 04:43:57.889304 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39759
I20260924 04:43:57.891848 14902 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
Sep 24 04:43:57 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:43:57.900243 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.888714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38931 (local address 127.0.0.1:56578)
0924 04:43:57.889324 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:57.889373 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:57.889511 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:57.890148 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:57.890154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:57.890591 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:57.890722 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.890730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.892148 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.892151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.892906 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:57.892916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.893505 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.893523 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:57.893632 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.894264 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:57.894279 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:57.896055 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:57.898318 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.898328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.898334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.898569 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.898880 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:57.898884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:57.898886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:57.898976 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:57.899522 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:57.899528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:57.899689 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 04:43:57.899852 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:43:57.900057 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":134,"threads_started":1}
I20260924 04:43:57.900254 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:57.888812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.1:38931)
0924 04:43:57.889675 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:57.889685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:57.889704 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:43:57.889763 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:57.889767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:57.889915 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:57.890063 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:57.890874 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.892019 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.893041 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:57.893375 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:57.893500 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:57.896252 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:57.896260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:57.896270 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:57.896292 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:57.898194 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.898690 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.898696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.898697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.898783 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:57.899075 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:57.899079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:57.899080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:57.899400 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:57.899549 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:57.899935 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 04:43:57.900093 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":113,"threads_started":1}
I20260924 04:43:57.901544 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa7c5640fae4902be2113d64f8099fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56578
I20260924 04:43:57.902555 14911 tablet_replica.cc:330] stopping tablet replica
I20260924 04:43:57.902863 14911 raft_consensus.cc:2229] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:57.903152 14911 raft_consensus.cc:2258] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:57.904136 14911 ts_tablet_manager.cc:1905] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:43:57.907742 14911 ts_tablet_manager.cc:1918] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:43:57.907857 14911 log.cc:1196] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/0aa7c5640fae4902be2113d64f8099fe
I20260924 04:43:57.908128 14911 ts_tablet_manager.cc:1939] T 0aa7c5640fae4902be2113d64f8099fe P c1d09b146dd642f191903131c90cecd2: Deleting consensus metadata
I20260924 04:43:57.909803 14701 catalog_manager.cc:4841] TS c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931): tablet 0aa7c5640fae4902be2113d64f8099fe (table test-table [id=18965779860d4d609070ec98648a0b93]) successfully deleted
W20260924 04:43:57.983398 14906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:57.983683 14906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:57.983729 14906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:57.987840 14906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:57.987901 14906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:57.988008 14906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:43:57.993371 14906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34679
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:57.994534 14906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:57.995697 14906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:58.005791 14916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:58.006033 14906 server_base.cc:1034] running on GCE node
W20260924 04:43:58.005808 14917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.005791 14919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:58.006728 14906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:58.007488 14906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:58.008723 14906 hybrid_clock.cc:648] HybridClock initialized: now 1790225038008700 us; error 35 us; skew 500 ppm
Sep 24 04:43:58 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:58.011930 14906 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:43:58.013068 14906 webserver.cc:460] Webserver started at http://127.0.98.2:34679/ using document root <none> and password file <none>
I20260924 04:43:58.013751 14906 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:58.013896 14906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:58.018258 14906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:43:58.020934 14926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.022488 14906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:43:58.022655 14906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "d82fb1b585434f0dadb086f4b2c8817e"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "e4e7a605794f257682cb844161c25f54"
server_key_iv: "930ec05e28ddbad0a6dcaf5ea1c14787"
server_key_version: "encryptionkey@0"
I20260924 04:43:58.023406 14906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:58.040916 14906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:58.044530 14906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:58.044767 14906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:58.045434 14906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:58.046343 14906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:58.046422 14906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.046501 14906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:58.046547 14906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.057185 14906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32905
I20260924 04:43:58.057221 15039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32905 every 8 connection(s)
I20260924 04:43:58.058372 14906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:43:58.067794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 14906
I20260924 04:43:58.068244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38529
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40439
--webserver_port=46275
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:43:58 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:58.074116 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.060645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41151 (local address 127.0.98.62:40439)
0924 04:43:58.060816 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.060820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.061860 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 04:43:58.063336 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.063339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.063406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.063536 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:58.064791 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.065333 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.066519 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.066768 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.066887 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.069648 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:58.069651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:58.069653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:58.069681 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:58.071773 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:58.072388 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:58.072392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:58.072393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:58.072475 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:58.072868 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:58.072872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:58.072874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:58.073076 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:58.073216 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.073779 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.073929 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:43:58.074999 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.061130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:41151)
0924 04:43:58.061663 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.061710 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.063080 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.063725 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:58.063732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:58.064117 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:58.064626 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.064640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.065469 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.065472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.066379 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.066388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.066928 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.066933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.067155 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.067874 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:58.067892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:58.069456 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:58.071916 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:58.071922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:58.071939 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:58.072247 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:58.072592 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:58.072595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:58.072597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:58.072761 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:58.073215 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:58.073223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:58.073496 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 04:43:58.073709 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 04:43:58.074102 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260924 04:43:58.076447 15040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260924 04:43:58.076747 15040 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:58.077458 15040 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260924 04:43:58.078912 14714 ts_manager.cc:194] Registered new tserver with Master: d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905)
I20260924 04:43:58.079744 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41151
W20260924 04:43:58.188832 15043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:58.189105 15043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:58.189152 15043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:58.193195 15043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:58.193264 15043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:58.193325 15043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:43:58.199312 15043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46275
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:58.200402 15043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:58.201364 15043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:58.211311 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.211308 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.211561 15049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:58.211727 15043 server_base.cc:1034] running on GCE node
I20260924 04:43:58.212335 15043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:58.213205 15043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:58.214414 15043 hybrid_clock.cc:648] HybridClock initialized: now 1790225038214380 us; error 58 us; skew 500 ppm
Sep 24 04:43:58 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:58.218365 15043 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:43:58.219448 15043 webserver.cc:460] Webserver started at http://127.0.98.3:46275/ using document root <none> and password file <none>
I20260924 04:43:58.220145 15043 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:58.220227 15043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:58.224506 15043 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 04:43:58.227480 15058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.229094 15043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:43:58.229261 15043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "4c947446296a4657a93d8d82e73db77b"
format_stamp: "Formatted at 2026-09-24 04:43:57 on dist-test-slave-npg2"
server_key: "6a1af69af4ae73711b18226a8669b6ee"
server_key_iv: "fb2bd3a6fe8afbd32b5815acdeaa5d2c"
server_key_version: "encryptionkey@0"
I20260924 04:43:58.229913 15043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:58.246079 15043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:58.249518 15043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:58.249744 15043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:58.250379 15043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:58.251281 15043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:58.251329 15043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.251395 15043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:58.251432 15043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.260898 15171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38529 every 8 connection(s)
I20260924 04:43:58.260918 15043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38529
I20260924 04:43:58.262138 15043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:43:58.265641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15043
Sep 24 04:43:58 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:43:58.275817 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.266394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.0.98.62:40439)
0924 04:43:58.266628 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.266633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.266692 (+ 59us) server_negotiation.cc:407] Connection header received
0924 04:43:58.266807 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.266810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.266862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.266965 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:58.267933 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.268752 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.269816 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.270091 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.270209 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.271835 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:58.271842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:58.271848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:58.271877 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:58.273839 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:58.274422 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:58.274425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:58.274427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:58.274538 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:58.274839 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:58.274842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:58.274843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:58.275028 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:58.275247 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.275455 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.275585 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260924 04:43:58.278978 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.264375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40423 (local address 127.0.98.62:40439)
0924 04:43:58.264522 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.264526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.265432 (+ 906us) server_negotiation.cc:407] Connection header received
0924 04:43:58.267481 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.267487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.267549 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.267695 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:43:58.269258 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.269947 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.271887 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.272127 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.272245 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.275350 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:43:58.275372 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:43:58.275378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:43:58.275404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:43:58.276854 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:58.277541 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:58.277544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:58.277545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:58.277654 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:43:58.278043 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:43:58.278046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:43:58.278048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:43:58.278204 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:43:58.278278 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.278773 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.278867 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":1920}
I20260924 04:43:58.279847 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.264714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:40423)
0924 04:43:58.265214 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.265258 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.267262 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.267857 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:58.267887 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:58.268474 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:43:58.269112 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.269127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.270152 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.270157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.271723 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.271758 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.272363 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.272367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.272582 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.273231 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:43:58.273255 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:43:58.275194 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:43:58.276977 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:58.276984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:58.277001 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:58.277398 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:58.277752 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:43:58.277755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:43:58.277756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:43:58.277939 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:43:58.278292 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:43:58.278298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:43:58.278529 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 04:43:58.278698 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 04:43:58.278956 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260924 04:43:58.280979 15172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260924 04:43:58.281265 15172 heartbeater.cc:464] Registering TS with master...
I20260924 04:43:58.281766 15172 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260924 04:43:58.282805 14714 ts_manager.cc:194] Registered new tserver with Master: 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
I20260924 04:43:58.283468 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40423
I20260924 04:43:58.289878 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:43:58.294752 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.290622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.0.98.62:40439)
0924 04:43:58.290788 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.290791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.290910 (+ 119us) server_negotiation.cc:407] Connection header received
0924 04:43:58.291051 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.291053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.291231 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.291317 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:43:58.291619 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.292115 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.293059 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.293306 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.293390 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.293756 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:43:58.294180 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:43:58.294276 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.294403 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.294455 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 04:43:58.295486 14714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:43:58.299069 14714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:43:58.307955 14836 tablet_service.cc:1467] Processing CreateTablet for tablet 5f88d20a428046329ed988c76baeabae (DEFAULT_TABLE table=test-table [id=f9b1437b602f4b518bcbb7b4e73c97f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:58.308773 14836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f88d20a428046329ed988c76baeabae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:58.311678 14794 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Bootstrap starting.
I20260924 04:43:58.312635 14794 tablet_bootstrap.cc:654] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:58.313057 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.306603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.0.1:48740)
0924 04:43:58.306985 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.307083 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.307222 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.308122 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:58.308125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:58.308141 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:58.308257 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.308266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.309752 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.309758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.311113 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.311148 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.312465 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.312469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.312627 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.312656 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:43:58.312748 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:43:58.312877 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":54}
I20260924 04:43:58.313117 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.306715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38529 (local address 127.0.0.1:57384)
0924 04:43:58.307440 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.307478 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.307584 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.308232 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:58.308236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:58.308257 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:58.308371 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.308378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.309989 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.309993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.311163 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.311171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.312401 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.312405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.312536 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.312664 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:43:58.312753 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:43:58.312873 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":67,"thread_start_us":121,"threads_started":1}
I20260924 04:43:58.313488 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.306779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.0.98.2:32905)
0924 04:43:58.307649 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.307662 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.307680 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:43:58.307741 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.307745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.307935 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.308424 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:58.308594 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.309616 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.311286 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.312336 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.312999 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.313156 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.313219 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.313336 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":128,"threads_started":1}
I20260924 04:43:58.313647 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.306942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.0.98.3:38529)
0924 04:43:58.307350 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.307365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.307597 (+ 232us) server_negotiation.cc:407] Connection header received
0924 04:43:58.307696 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.307703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.307936 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.308110 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:58.308526 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.309838 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.311286 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.312259 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.312393 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.312565 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.313363 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.313474 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
I20260924 04:43:58.314400 14794 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: No bootstrap required, opened a new log
I20260924 04:43:58.314483 14794 ts_tablet_manager.cc:1397] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:43:58.314960 14794 raft_consensus.cc:348] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.315070 14794 raft_consensus.cc:374] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:58.315106 14794 raft_consensus.cc:729] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d09b146dd642f191903131c90cecd2, State: Initialized, Role: FOLLOWER
I20260924 04:43:58.315294 14794 consensus_queue.cc:260] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.316749 14794 ts_tablet_manager.cc:1428] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:43:58.317472 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 5f88d20a428046329ed988c76baeabae (DEFAULT_TABLE table=test-table [id=f9b1437b602f4b518bcbb7b4e73c97f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:58.317591 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 5f88d20a428046329ed988c76baeabae (DEFAULT_TABLE table=test-table [id=f9b1437b602f4b518bcbb7b4e73c97f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:43:58.318701 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f88d20a428046329ed988c76baeabae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:58.318748 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f88d20a428046329ed988c76baeabae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:43:58.325431 15185 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Bootstrap starting.
I20260924 04:43:58.325551 15186 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Bootstrap starting.
I20260924 04:43:58.328040 15185 tablet_bootstrap.cc:654] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:58.328555 15186 tablet_bootstrap.cc:654] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Neither blocks nor log segments found. Creating new log.
I20260924 04:43:58.329214 15185 log.cc:824] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:58.329257 15186 log.cc:824] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:58.330976 15185 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: No bootstrap required, opened a new log
I20260924 04:43:58.331043 15186 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: No bootstrap required, opened a new log
I20260924 04:43:58.331216 15185 ts_tablet_manager.cc:1397] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:43:58.331257 15186 ts_tablet_manager.cc:1397] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:43:58.334569 15185 raft_consensus.cc:348] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.334810 15185 raft_consensus.cc:374] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:58.334844 15185 raft_consensus.cc:729] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c947446296a4657a93d8d82e73db77b, State: Initialized, Role: FOLLOWER
I20260924 04:43:58.334719 15186 raft_consensus.cc:348] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.335069 15186 raft_consensus.cc:374] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:43:58.335155 15186 raft_consensus.cc:729] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82fb1b585434f0dadb086f4b2c8817e, State: Initialized, Role: FOLLOWER
I20260924 04:43:58.335362 15185 consensus_queue.cc:260] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.335641 15186 consensus_queue.cc:260] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.336086 15172 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260924 04:43:58.336320 15040 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260924 04:43:58.336472 15185 ts_tablet_manager.cc:1428] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:43:58.336684 15186 ts_tablet_manager.cc:1428] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260924 04:43:58.357858 14903 tablet.cc:2369] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:43:58.506242 14908 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:43:58.506489 14908 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.507795 14908 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
I20260924 04:43:58.512315 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.508087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38529 (local address 127.0.98.1:40795)
0924 04:43:58.508451 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.508464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.508590 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.508854 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:58.508857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:58.508873 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:58.509038 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.509043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.509774 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.509779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.511172 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.511183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.511846 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.511848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.511926 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.511964 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:43:58.512066 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:43:58.512145 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":130,"thread_start_us":76,"threads_started":1}
I20260924 04:43:58.512328 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.508273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40795 (local address 127.0.98.3:38529)
0924 04:43:58.508400 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.508405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.508539 (+ 134us) server_negotiation.cc:407] Connection header received
0924 04:43:58.508703 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.508708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.508756 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.508842 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:58.509153 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.509659 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.511304 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.511721 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.511810 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.511863 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.512087 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.512227 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:43:58.512862 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.508141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40991 (local address 127.0.98.2:32905)
0924 04:43:58.508271 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.508276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.508430 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:43:58.508639 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.508643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.508695 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.508786 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:43:58.509102 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.509608 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.511914 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.512318 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.512438 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.512494 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.512672 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.512741 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:43:58.512910 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.507975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.98.1:40991)
0924 04:43:58.508323 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.508346 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.508500 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.508797 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:43:58.508800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:43:58.508816 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:43:58.508984 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.508990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.509727 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.509731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.511178 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:43:58.511183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.512442 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.512445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:43:58.512534 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.512555 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:43:58.512658 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:43:58.512730 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":145,"thread_start_us":104,"threads_started":1}
W20260924 04:43:58.513070 15173 tablet.cc:2369] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:43:58.513625 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f88d20a428046329ed988c76baeabae" candidate_uuid: "c1d09b146dd642f191903131c90cecd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82fb1b585434f0dadb086f4b2c8817e" is_pre_election: true
I20260924 04:43:58.513756 15126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f88d20a428046329ed988c76baeabae" candidate_uuid: "c1d09b146dd642f191903131c90cecd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c947446296a4657a93d8d82e73db77b" is_pre_election: true
I20260924 04:43:58.514016 14994 raft_consensus.cc:2454] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d09b146dd642f191903131c90cecd2 in term 0.
I20260924 04:43:58.514102 15126 raft_consensus.cc:2454] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d09b146dd642f191903131c90cecd2 in term 0.
I20260924 04:43:58.514485 14791 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1d09b146dd642f191903131c90cecd2, d82fb1b585434f0dadb086f4b2c8817e; no voters:
I20260924 04:43:58.514783 14908 raft_consensus.cc:2781] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:43:58.514873 14908 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:43:58.514921 14908 raft_consensus.cc:3037] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:58.515926 14908 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.516300 14908 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 1 election: Requested vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
I20260924 04:43:58.516687 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f88d20a428046329ed988c76baeabae" candidate_uuid: "c1d09b146dd642f191903131c90cecd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82fb1b585434f0dadb086f4b2c8817e"
I20260924 04:43:58.516816 14994 raft_consensus.cc:3037] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:58.516855 15126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f88d20a428046329ed988c76baeabae" candidate_uuid: "c1d09b146dd642f191903131c90cecd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c947446296a4657a93d8d82e73db77b"
I20260924 04:43:58.516985 15126 raft_consensus.cc:3037] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:43:58.517944 14994 raft_consensus.cc:2454] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d09b146dd642f191903131c90cecd2 in term 1.
I20260924 04:43:58.518075 15126 raft_consensus.cc:2454] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d09b146dd642f191903131c90cecd2 in term 1.
I20260924 04:43:58.518239 14791 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1d09b146dd642f191903131c90cecd2, d82fb1b585434f0dadb086f4b2c8817e; no voters:
I20260924 04:43:58.518422 14908 raft_consensus.cc:2781] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:43:58.518626 14908 raft_consensus.cc:686] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 LEADER]: Becoming Leader. State: Replica: c1d09b146dd642f191903131c90cecd2, State: Running, Role: LEADER
I20260924 04:43:58.518890 14908 consensus_queue.cc:237] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:58.522008 14714 catalog_manager.cc:5495] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d09b146dd642f191903131c90cecd2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c1d09b146dd642f191903131c90cecd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } health_report { overall_health: HEALTHY } } }
W20260924 04:43:58.559671 15041 tablet.cc:2369] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:43:58.559806 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:58.556127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.0.98.1:38931)
0924 04:43:58.556386 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 04:43:58.556393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:43:58.556406 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:43:58.556480 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:43:58.556482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:43:58.556546 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:43:58.556647 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:43:58.556876 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.557443 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.558264 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:43:58.558482 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:43:58.558555 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:43:58.558787 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:43:58.559322 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:43:58.559411 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:43:58.559532 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:43:58.559580 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
I20260924 04:43:58.563954 14994 raft_consensus.cc:1261] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Refusing update from remote peer c1d09b146dd642f191903131c90cecd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:58.563954 15126 raft_consensus.cc:1261] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Refusing update from remote peer c1d09b146dd642f191903131c90cecd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:43:58.564771 15193 consensus_queue.cc:1035] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:58.564927 14908 consensus_queue.cc:1035] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:43:58.571898 15197 mvcc.cc:205] Tried to move back new op lower bound from 7332761757950844928 to 7332761757776543744. Current Snapshot: MvccSnapshot[applied={T|T < 7332761757950844928}]
I20260924 04:43:58.572429 15198 mvcc.cc:205] Tried to move back new op lower bound from 7332761757950844928 to 7332761757776543744. Current Snapshot: MvccSnapshot[applied={T|T < 7332761757950844928}]
I20260924 04:43:58.574741 15199 mvcc.cc:205] Tried to move back new op lower bound from 7332761757950844928 to 7332761757776543744. Current Snapshot: MvccSnapshot[applied={T|T < 7332761757950844928}]
I20260924 04:43:58.581214 14713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:43:58.581377 14713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:43:58.584283 14713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Sending DeleteTablet for 3 replicas of tablet 5f88d20a428046329ed988c76baeabae
I20260924 04:43:58.585199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14752
I20260924 04:43:58.585260 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f88d20a428046329ed988c76baeabae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48740
I20260924 04:43:58.585975 15106 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f88d20a428046329ed988c76baeabae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57384
I20260924 04:43:58.586524 15208 tablet_replica.cc:330] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: stopping tablet replica
I20260924 04:43:58.586808 15207 tablet_replica.cc:330] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: stopping tablet replica
I20260924 04:43:58.586915 15208 raft_consensus.cc:2229] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:58.587185 15207 raft_consensus.cc:2229] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:43:58.587371 15208 raft_consensus.cc:2258] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:58.587540 15207 raft_consensus.cc:2258] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:43:58.588999 15208 ts_tablet_manager.cc:1905] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:43:58.589463 15207 ts_tablet_manager.cc:1905] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:43:58.591495 15208 ts_tablet_manager.cc:1918] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:43:58.591629 15208 log.cc:1196] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/5f88d20a428046329ed988c76baeabae
I20260924 04:43:58.591924 15208 ts_tablet_manager.cc:1939] T 5f88d20a428046329ed988c76baeabae P 4c947446296a4657a93d8d82e73db77b: Deleting consensus metadata
I20260924 04:43:58.592947 15207 ts_tablet_manager.cc:1918] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:43:58.593048 15207 log.cc:1196] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/5f88d20a428046329ed988c76baeabae
I20260924 04:43:58.593173 14700 catalog_manager.cc:4841] TS 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): tablet 5f88d20a428046329ed988c76baeabae (table test-table [id=f9b1437b602f4b518bcbb7b4e73c97f4]) successfully deleted
I20260924 04:43:58.593379 15207 ts_tablet_manager.cc:1939] T 5f88d20a428046329ed988c76baeabae P d82fb1b585434f0dadb086f4b2c8817e: Deleting consensus metadata
W20260924 04:43:58.594724 14701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38931 (error 108)
I20260924 04:43:58.595287 14701 catalog_manager.cc:4841] TS d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): tablet 5f88d20a428046329ed988c76baeabae (table test-table [id=f9b1437b602f4b518bcbb7b4e73c97f4]) successfully deleted
W20260924 04:43:58.596084 15181 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:43:58.595598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38931 (local address 127.0.0.1:56600)
0924 04:43:58.595731 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:58.595750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:58.595861 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:58.595958 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38931: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 04:43:58.596290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14906
W20260924 04:43:58.596247 14701 catalog_manager.cc:4584] TS c1d09b146dd642f191903131c90cecd2 (127.0.98.1:38931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f88d20a428046329ed988c76baeabae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38931: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 04:43:58.602624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 15043
I20260924 04:43:58.608479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 14681
I20260924 04:43:58.615080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--webserver_interface=127.0.98.62
--webserver_port=35915
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:43:58.737950 15210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:58.738330 15210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:58.738423 15210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:58.743184 15210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:43:58.743283 15210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:58.743327 15210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:58.743388 15210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:43:58.743425 15210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:43:58.749578 15210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:58.751015 15210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:58.752266 15210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:43:58.762089 15210 server_base.cc:1034] running on GCE node
W20260924 04:43:58.762034 15215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.762189 15216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.762037 15218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:58.762904 15210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:58.764143 15210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:58.765347 15210 hybrid_clock.cc:648] HybridClock initialized: now 1790225038765331 us; error 31 us; skew 500 ppm
Sep 24 04:43:58 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:58.768283 15210 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:43:58.769238 15210 webserver.cc:460] Webserver started at http://127.0.98.62:35915/ using document root <none> and password file <none>
I20260924 04:43:58.769865 15210 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:58.769963 15210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:58.774134 15210 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:43:58.776539 15225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.777894 15210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:43:58.778038 15210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "31e5c0e1aebe4647ab496aad06aabf04"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "d0ff7be98f02b38df4ed5c07b487a2b9"
server_key_iv: "dd9d270db9f6ec2931ebe2887f7e6adf"
server_key_version: "encryptionkey@0"
I20260924 04:43:58.778625 15210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:58.788923 15210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:58.792304 15210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:58.792518 15210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:58.799880 15210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40439
I20260924 04:43:58.799927 15277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40439 every 8 connection(s)
I20260924 04:43:58.800981 15210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:43:58.802385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15210
I20260924 04:43:58.802855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38931
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40439
--webserver_port=36161
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:58.806509 15278 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:43:58.809357 15278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap starting.
I20260924 04:43:58.833585 15278 log.cc:824] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:58.847918 15278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:43:58.848453 15278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04: Bootstrap complete.
I20260924 04:43:58.853539 15278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:58.854000 15278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e5c0e1aebe4647ab496aad06aabf04, State: Initialized, Role: FOLLOWER
I20260924 04:43:58.854593 15278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:58.854746 15278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:43:58.854836 15278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:43:58.854962 15278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:43:58.857038 15278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:58.857410 15278 leader_election.cc:302] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e5c0e1aebe4647ab496aad06aabf04; no voters:
I20260924 04:43:58.857781 15278 leader_election.cc:288] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:43:58.857908 15282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:43:58.858186 15282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [term 3 LEADER]: Becoming Leader. State: Replica: 31e5c0e1aebe4647ab496aad06aabf04, State: Running, Role: LEADER
I20260924 04:43:58.858491 15282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260924 04:43:58.859016 15278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:43:58.859519 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e5c0e1aebe4647ab496aad06aabf04. Latest consensus state: current_term: 3 leader_uuid: "31e5c0e1aebe4647ab496aad06aabf04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260924 04:43:58.859630 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:58.859956 15286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:43:58.860113 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "31e5c0e1aebe4647ab496aad06aabf04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e5c0e1aebe4647ab496aad06aabf04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260924 04:43:58.860288 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e5c0e1aebe4647ab496aad06aabf04 [sys.catalog]: This master's current role is: LEADER
I20260924 04:43:58.865701 15286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:43:58.866340 15286 catalog_manager.cc:1259] Loaded cluster ID: 669dbe67049246158c7f5dce652b6642
I20260924 04:43:58.866406 15286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 04:43:58.867381 15286 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa2b2c2e9ad google::LogMessage::Fail() at ??:0
@ 0x7fa2b2c3188a google::LogMessage::SendToLog() at ??:0
@ 0x7fa2b2c2e4de google::LogMessage::Flush() at ??:0
@ 0x7fa2b2c2fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa2bd56dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa2bd56ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa2bd56b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa2bd5af640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa2b4e803d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa2b4e80c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa2b4e823f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa2b4e71512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2b64b56db start_thread at ??:0
@ 0x7fa2b172c61f clone at ??:0
W20260924 04:43:58.956080 15280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:58.956400 15280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:58.956494 15280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:58.960775 15280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:58.960858 15280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:58.960948 15280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:43:58.966372 15280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36161
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:58.967592 15280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:58.968637 15280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:58.978541 15322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.978564 15319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:58.978543 15320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:58.978789 15280 server_base.cc:1034] running on GCE node
I20260924 04:43:58.979393 15280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:58.980180 15280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:58.981375 15280 hybrid_clock.cc:648] HybridClock initialized: now 1790225038981351 us; error 34 us; skew 500 ppm
Sep 24 04:43:58 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:58.984773 15280 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:43:58.985883 15280 webserver.cc:460] Webserver started at http://127.0.98.1:36161/ using document root <none> and password file <none>
I20260924 04:43:58.986505 15280 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:58.986599 15280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:58.990980 15280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:43:58.993422 15329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:58.994839 15280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:43:58.994989 15280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "c1d09b146dd642f191903131c90cecd2"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "94be01fc365327147495e1a9b4f1111d"
server_key_iv: "82aad18e6974218427a18d19481fbad4"
server_key_version: "encryptionkey@0"
I20260924 04:43:58.995654 15280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:59.029349 15280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:59.033236 15280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:59.033483 15280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:59.034209 15280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:59.035487 15336 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:43:59.041105 15280 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:43:59.041218 15280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:43:59.041316 15280 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:43:59.043110 15280 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:43:59.043253 15280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260924 04:43:59.043534 15336 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Bootstrap starting.
I20260924 04:43:59.054490 15280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38931
I20260924 04:43:59.054759 15443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38931 every 8 connection(s)
I20260924 04:43:59.055688 15280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:43:59.061234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15280
I20260924 04:43:59.061684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32905
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40439
--webserver_port=34679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:59.065840 15336 log.cc:824] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:43:59.076589 15336 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:43:59.077029 15336 tablet_bootstrap.cc:492] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Bootstrap complete.
I20260924 04:43:59.078465 15336 ts_tablet_manager.cc:1397] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.016s
I20260924 04:43:59.081796 15336 raft_consensus.cc:348] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:59.082392 15336 raft_consensus.cc:729] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d09b146dd642f191903131c90cecd2, State: Initialized, Role: FOLLOWER
I20260924 04:43:59.082979 15336 consensus_queue.cc:260] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:43:59.084195 15336 ts_tablet_manager.cc:1428] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260924 04:43:59.119058 15446 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:43:59.057823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:46569)
0924 04:43:59.058337 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.058393 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:43:59.059344 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:43:59.118176 (+ 58832us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":169,"threads_started":1}
W20260924 04:43:59.119720 15444 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40439 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40439: Client connection negotiation failed: client connection to 127.0.98.62:40439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:43:59.120575 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.120284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:44045)
0924 04:43:59.120416 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.120494 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:43:59.121547 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.121309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:41907)
0924 04:43:59.121440 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.121498 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 04:43:59.121752 15444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:43:59.189138 15447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:59.189459 15447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:59.189553 15447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:59.193667 15447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:59.193763 15447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:59.193857 15447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:43:59.199477 15447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34679
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:59.200670 15447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:59.201738 15447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:43:59.211990 15447 server_base.cc:1034] running on GCE node
W20260924 04:43:59.211928 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:59.211928 15455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:59.211951 15457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:59.212648 15447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:59.213357 15447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:59.214563 15447 hybrid_clock.cc:648] HybridClock initialized: now 1790225039214528 us; error 47 us; skew 500 ppm
Sep 24 04:43:59 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:59.217577 15447 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:43:59.218526 15447 webserver.cc:460] Webserver started at http://127.0.98.2:34679/ using document root <none> and password file <none>
I20260924 04:43:59.219085 15447 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:59.219210 15447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:59.223330 15447 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:43:59.225574 15464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:59.226873 15447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:43:59.227034 15447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "d82fb1b585434f0dadb086f4b2c8817e"
format_stamp: "Formatted at 2026-09-24 04:43:56 on dist-test-slave-npg2"
server_key: "e4e7a605794f257682cb844161c25f54"
server_key_iv: "930ec05e28ddbad0a6dcaf5ea1c14787"
server_key_version: "encryptionkey@0"
I20260924 04:43:59.227658 15447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:59.244921 15447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:59.248662 15447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:59.248842 15447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:59.249433 15447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:59.250253 15447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:59.250304 15447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:59.250351 15447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:59.250367 15447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:59.259894 15447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32905
I20260924 04:43:59.259922 15577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32905 every 8 connection(s)
I20260924 04:43:59.260978 15447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:43:59.261214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15447
I20260924 04:43:59.261700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38529
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40439
--webserver_port=46275
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:43:59.265417 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.263422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:60303)
0924 04:43:59.264035 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.264204 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":157,"threads_started":1}
W20260924 04:43:59.266057 15578 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40439 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40439: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
I20260924 04:43:59.266836 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.266516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:57171)
0924 04:43:59.266672 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.266746 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:43:59.267748 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.267455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:57115)
0924 04:43:59.267599 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.267671 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 04:43:59.267954 15578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:43:59.380100 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:43:59.380409 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:43:59.380456 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:43:59.384770 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:43:59.384855 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:43:59.384956 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:43:59.390818 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46275
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:43:59.392047 15580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:43:59.393191 15580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:43:59.403867 15586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:59.403855 15589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:43:59.403869 15587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:43:59.403910 15580 server_base.cc:1034] running on GCE node
I20260924 04:43:59.404593 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:43:59.405287 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:43:59.406455 15580 hybrid_clock.cc:648] HybridClock initialized: now 1790225039406438 us; error 34 us; skew 500 ppm
Sep 24 04:43:59 dist-test-slave-npg2 krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:43:59.409544 15580 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:43:59.410521 15580 webserver.cc:460] Webserver started at http://127.0.98.3:46275/ using document root <none> and password file <none>
I20260924 04:43:59.411279 15580 fs_manager.cc:359] Metadata directory not provided
I20260924 04:43:59.411381 15580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:43:59.415922 15580 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:43:59.418157 15596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:59.419560 15580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:43:59.419703 15580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "4c947446296a4657a93d8d82e73db77b"
format_stamp: "Formatted at 2026-09-24 04:43:57 on dist-test-slave-npg2"
server_key: "6a1af69af4ae73711b18226a8669b6ee"
server_key_iv: "fb2bd3a6fe8afbd32b5815acdeaa5d2c"
server_key_version: "encryptionkey@0"
I20260924 04:43:59.420325 15580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:43:59.430382 15580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:43:59.433768 15580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:43:59.434005 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:43:59.434644 15580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:43:59.435545 15580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:43:59.435627 15580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:59.435701 15580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:43:59.435752 15580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:43:59.445422 15580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38529
I20260924 04:43:59.445468 15709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38529 every 8 connection(s)
I20260924 04:43:59.446476 15580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:43:59.450615 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.448729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:53925)
0924 04:43:59.449221 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.449333 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260924 04:43:59.450729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15580
W20260924 04:43:59.451242 15710 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40439 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40439: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
I20260924 04:43:59.452013 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.451687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:40131)
0924 04:43:59.451873 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.451938 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 04:43:59.452726 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:43:59.452519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:38841)
0924 04:43:59.452632 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:43:59.452675 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:43:59.453022 15710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:44:00.122826 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.122365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:45845)
0924 04:44:00.122640 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:00.122707 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260924 04:44:00.269008 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.268568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:40547)
0924 04:44:00.268834 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:00.268895 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260924 04:44:00.454155 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.453723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:49267)
0924 04:44:00.453986 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:00.454048 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260924 04:44:00.498549 15717 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:00.498798 15717 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:44:00.500133 15717 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
Sep 24 04:44:00 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:44:00 dist-test-slave-npg2 krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:44:00.516393 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.500503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.98.1:41467)
0924 04:44:00.500850 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:00.500862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:00.500939 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:00.502651 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:00.502657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:00.503323 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:00.503855 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:00.503872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.505228 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.505231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:00.506237 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:00.506249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.507249 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.507253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:00.507436 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:00.508309 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:00.508337 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:00.510858 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:00.513647 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:00.513650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:00.513652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:00.513974 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:00.514540 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:00.514543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:00.514545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:00.514672 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:00.515517 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:00.515523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:00.515742 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 04:44:00.515918 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 04:44:00.516241 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":177,"spinlock_wait_cycles":5760,"thread_start_us":75,"threads_started":1}
I20260924 04:44:00.516577 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.500465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38529 (local address 127.0.98.1:53037)
0924 04:44:00.500730 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:00.500748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:00.500861 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:00.502651 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:00.502657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:00.503323 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:00.503854 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:00.503872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.505224 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.505229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:00.506245 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:00.506255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.507307 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.507311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:00.507437 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:00.508297 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:00.508320 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:00.510458 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:00.513423 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:00.513434 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:00.513450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:00.513829 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:00.514326 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:00.514329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:00.514331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:00.514539 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:00.515441 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:00.515449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:00.515754 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 04:44:00.515951 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 04:44:00.516356 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":318,"spinlock_wait_cycles":11264}
I20260924 04:44:00.516665 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.500651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41467 (local address 127.0.98.2:32905)
0924 04:44:00.501089 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:00.501110 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:00.501151 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:44:00.501841 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:00.501847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:00.502232 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:00.502579 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:00.504063 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.505068 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.506420 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.507035 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.507301 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:00.511261 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:00.511266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:00.511278 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:00.511311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:00.513508 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:00.514213 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:00.514218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:00.514222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:00.514426 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:00.514780 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:00.514783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:00.514784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:00.515319 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:00.515467 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:00.516105 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 04:44:00.516459 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":71,"threads_started":1}
I20260924 04:44:00.517510 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:00.500656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53037 (local address 127.0.98.3:38529)
0924 04:44:00.501190 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:00.501209 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:00.501243 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:44:00.501880 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:00.501886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:00.502232 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:00.502578 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:00.504114 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.505066 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.506455 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:00.507023 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:00.507377 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:00.510816 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:00.510824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:00.510843 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:00.510885 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:00.513262 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:00.513981 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:00.513989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:00.513997 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:00.514199 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:00.514645 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:00.514649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:00.514651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:00.515313 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:00.515461 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:00.517052 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0924 04:44:00.517324 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":140,"threads_started":1}
W20260924 04:44:00.519618 15333 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
W20260924 04:44:00.520200 15332 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
I20260924 04:44:00.520308 15332 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d09b146dd642f191903131c90cecd2; no voters: 4c947446296a4657a93d8d82e73db77b, d82fb1b585434f0dadb086f4b2c8817e
I20260924 04:44:00.520671 15717 raft_consensus.cc:2726] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:44:01.124069 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:01.123624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:40189)
0924 04:44:01.123907 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:01.123969 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260924 04:44:01.270553 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:01.270099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:50159)
0924 04:44:01.270386 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:01.270446 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260924 04:44:01.455574 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:01.455059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:33101)
0924 04:44:01.455404 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:01.455476 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":140,"threads_started":1}
I20260924 04:44:02.123245 15724 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:02.123445 15724 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:44:02.124213 15724 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
I20260924 04:44:02.125379 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:02.124859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:49285)
0924 04:44:02.125211 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:02.125289 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":79,"threads_started":1}
W20260924 04:44:02.125550 15333 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
W20260924 04:44:02.125671 15332 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
I20260924 04:44:02.125775 15332 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d09b146dd642f191903131c90cecd2; no voters: 4c947446296a4657a93d8d82e73db77b, d82fb1b585434f0dadb086f4b2c8817e
I20260924 04:44:02.125995 15724 raft_consensus.cc:2726] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:44:02.272047 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:02.271560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:38595)
0924 04:44:02.271835 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:02.271936 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
I20260924 04:44:02.456969 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:02.456436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:48763)
0924 04:44:02.456711 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:02.456775 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260924 04:44:03.126843 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:03.126291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:55513)
0924 04:44:03.126645 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:03.126725 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260924 04:44:03.273586 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:03.273070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:44421)
0924 04:44:03.273400 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:03.273479 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260924 04:44:03.458376 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:03.457841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:42681)
0924 04:44:03.458201 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:03.458264 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":88,"threads_started":1}
I20260924 04:44:03.671810 15731 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:03.671947 15731 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:44:03.672428 15731 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
W20260924 04:44:03.673278 15333 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
W20260924 04:44:03.673393 15332 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
I20260924 04:44:03.673516 15332 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d09b146dd642f191903131c90cecd2; no voters: 4c947446296a4657a93d8d82e73db77b, d82fb1b585434f0dadb086f4b2c8817e
I20260924 04:44:03.673703 15731 raft_consensus.cc:2726] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:44:04.128712 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:04.128151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:44749)
0924 04:44:04.128549 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:04.128611 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":115,"threads_started":1}
I20260924 04:44:04.275074 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:04.274597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:39811)
0924 04:44:04.274901 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:04.274969 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260924 04:44:04.460078 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:04.459613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:33913)
0924 04:44:04.459909 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:04.459976 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260924 04:44:05.130214 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:05.129747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:35853)
0924 04:44:05.130049 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:05.130113 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
I20260924 04:44:05.276588 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:05.276098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:36399)
0924 04:44:05.276408 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:05.276483 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260924 04:44:05.461417 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:05.460999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:37413)
0924 04:44:05.461261 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:05.461322 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260924 04:44:05.760635 15738 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:05.760888 15738 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:44:05.761340 15738 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
W20260924 04:44:05.762358 15333 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
W20260924 04:44:05.762454 15332 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
I20260924 04:44:05.762568 15332 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d09b146dd642f191903131c90cecd2; no voters: 4c947446296a4657a93d8d82e73db77b, d82fb1b585434f0dadb086f4b2c8817e
I20260924 04:44:05.762812 15738 raft_consensus.cc:2726] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:44:06.131814 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:06.131251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:41995)
0924 04:44:06.131550 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:06.131683 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260924 04:44:06.278153 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:06.277623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:52551)
0924 04:44:06.277971 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:06.278039 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":93,"threads_started":1}
I20260924 04:44:06.462962 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:06.462410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:43505)
0924 04:44:06.462795 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:06.462865 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":77,"threads_started":1}
I20260924 04:44:07.133407 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:07.132828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:50827)
0924 04:44:07.133145 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:07.133228 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260924 04:44:07.279768 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:07.279059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:42659)
0924 04:44:07.279524 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:07.279582 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":296,"threads_started":1}
I20260924 04:44:07.464541 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:07.464016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:38149)
0924 04:44:07.464349 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:07.464438 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260924 04:44:08.134850 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:08.134330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:47891)
0924 04:44:08.134671 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:08.134746 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260924 04:44:08.281198 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:08.280619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:35831)
0924 04:44:08.280949 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:08.281024 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260924 04:44:08.465902 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:08.465448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:36261)
0924 04:44:08.465720 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:08.465803 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260924 04:44:09.136446 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:09.135935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:43259)
0924 04:44:09.136236 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:09.136304 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260924 04:44:09.282582 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:09.282145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:52219)
0924 04:44:09.282408 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:09.282470 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260924 04:44:09.467329 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:09.466826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:45793)
0924 04:44:09.467118 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:09.467232 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260924 04:44:10.137681 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:10.137234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:58143)
0924 04:44:10.137505 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:10.137573 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260924 04:44:10.283867 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:10.283381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:33959)
0924 04:44:10.283694 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:10.283766 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260924 04:44:10.468577 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:10.468180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:43847)
0924 04:44:10.468428 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:10.468481 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260924 04:44:10.653051 15754 raft_consensus.cc:482] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:10.653187 15754 raft_consensus.cc:504] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82fb1b585434f0dadb086f4b2c8817e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "4c947446296a4657a93d8d82e73db77b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38529 } } peers { permanent_uuid: "c1d09b146dd642f191903131c90cecd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38931 } }
I20260924 04:44:10.653841 15754 leader_election.cc:288] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905), 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529)
W20260924 04:44:10.654639 15333 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82fb1b585434f0dadb086f4b2c8817e (127.0.98.2:32905): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
W20260924 04:44:10.654750 15332 leader_election.cc:341] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c947446296a4657a93d8d82e73db77b (127.0.98.3:38529): Not found: Tablet not found: 5f88d20a428046329ed988c76baeabae
I20260924 04:44:10.654865 15332 leader_election.cc:302] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d09b146dd642f191903131c90cecd2; no voters: 4c947446296a4657a93d8d82e73db77b, d82fb1b585434f0dadb086f4b2c8817e
I20260924 04:44:10.654999 15754 raft_consensus.cc:2726] T 5f88d20a428046329ed988c76baeabae P c1d09b146dd642f191903131c90cecd2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:44:11.139111 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:11.138626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:50015)
0924 04:44:11.138903 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:11.138973 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260924 04:44:11.285458 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:11.284949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:60025)
0924 04:44:11.285291 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:11.285355 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260924 04:44:11.469918 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:11.469502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:40267)
0924 04:44:11.469762 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:11.469817 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260924 04:44:12.140460 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:12.139995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:36845)
0924 04:44:12.140293 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:12.140362 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260924 04:44:12.286911 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:12.286439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:59697)
0924 04:44:12.286743 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:12.286808 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260924 04:44:12.471383 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:12.470887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:51213)
0924 04:44:12.471220 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:12.471284 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260924 04:44:13.141871 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:13.141428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:58905)
0924 04:44:13.141705 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:13.141761 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
I20260924 04:44:13.288537 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:13.287899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:34197)
0924 04:44:13.288304 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:13.288409 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":316,"threads_started":1}
I20260924 04:44:13.473024 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:13.472459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:51567)
0924 04:44:13.472840 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:13.472912 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":146,"threads_started":1}
I20260924 04:44:14.143501 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:14.142897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:60043)
0924 04:44:14.143334 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:14.143403 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":87,"threads_started":1}
I20260924 04:44:14.290010 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:14.289528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:42369)
0924 04:44:14.289832 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:14.289898 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
I20260924 04:44:14.460729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 15280
I20260924 04:44:14.466756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 15447
I20260924 04:44:14.472182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 15580
2026-09-24T04:44:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19358 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:44:14 dist-test-slave-npg2 krb5kdc[15768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:44:14 dist-test-slave-npg2 krb5kdc[15768](info): set up 2 sockets
Sep 24 04:44:14 dist-test-slave-npg2 krb5kdc[15768](info): commencing operation
krb5kdc: starting...
W20260924 04:44:15.547541 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:44:15 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:44:15.674626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:44:15.790303 15781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:15.790603 15781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:15.790647 15781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:15.794878 15781 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:44:15.794952 15781 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:44:15.794968 15781 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:44:15.794983 15781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:44:15.795002 15781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:15.795020 15781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:15.795037 15781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:44:15.795053 15781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:44:15.800700 15781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:15.801784 15781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:15.802783 15781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:15.812161 15789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:15.812107 15787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:15.812107 15786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:15.812584 15781 server_base.cc:1034] running on GCE node
I20260924 04:44:15.813334 15781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:15.814610 15781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:15.815807 15781 hybrid_clock.cc:648] HybridClock initialized: now 1790225055815720 us; error 110 us; skew 500 ppm
Sep 24 04:44:15 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:15.819039 15781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:44:15.820077 15781 webserver.cc:460] Webserver started at http://127.0.98.62:37263/ using document root <none> and password file <none>
I20260924 04:44:15.820674 15781 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:15.820767 15781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:15.821013 15781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:44:15.823642 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "34a89057b31f4a3fa635de52a272bd34"
format_stamp: "Formatted at 2026-09-24 04:44:15 on dist-test-slave-npg2"
server_key: "973530e5a9d5e9a28c0d2be6b8f38f0a"
server_key_iv: "cff2f7ca017289c386a6c94278ce3f89"
server_key_version: "encryptionkey@0"
I20260924 04:44:15.824234 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "34a89057b31f4a3fa635de52a272bd34"
format_stamp: "Formatted at 2026-09-24 04:44:15 on dist-test-slave-npg2"
server_key: "973530e5a9d5e9a28c0d2be6b8f38f0a"
server_key_iv: "cff2f7ca017289c386a6c94278ce3f89"
server_key_version: "encryptionkey@0"
I20260924 04:44:15.828382 15781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:44:15.831496 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:15.832978 15781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:15.833151 15781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "34a89057b31f4a3fa635de52a272bd34"
format_stamp: "Formatted at 2026-09-24 04:44:15 on dist-test-slave-npg2"
server_key: "973530e5a9d5e9a28c0d2be6b8f38f0a"
server_key_iv: "cff2f7ca017289c386a6c94278ce3f89"
server_key_version: "encryptionkey@0"
I20260924 04:44:15.833299 15781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:15.861263 15781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:15.864964 15781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:15.865206 15781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:15.873042 15781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44145
I20260924 04:44:15.873356 15848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44145 every 8 connection(s)
I20260924 04:44:15.874153 15781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:44:15.877189 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:15.881978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15781
I20260924 04:44:15.882148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:44:15.882500 392 external_mini_cluster.cc:1428] Setting key bd1f1acf83ffc388a62701cc92d9a520
I20260924 04:44:15.883605 15849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap starting.
I20260924 04:44:15.886554 15849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:15.887593 15849 log.cc:824] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:15.890000 15849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: No bootstrap required, opened a new log
Sep 24 04:44:15 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:15.895052 15849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:15.895408 15849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:15.895483 15849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a89057b31f4a3fa635de52a272bd34, State: Initialized, Role: FOLLOWER
I20260924 04:44:15.896049 15849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:15.896236 15849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:44:15.896317 15849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:44:15.896425 15849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:15.897634 15849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:15.898026 15849 leader_election.cc:302] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34a89057b31f4a3fa635de52a272bd34; no voters:
I20260924 04:44:15.898419 15849 leader_election.cc:288] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:44:15.898536 15854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:44:15.898874 15854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 LEADER]: Becoming Leader. State: Replica: 34a89057b31f4a3fa635de52a272bd34, State: Running, Role: LEADER
I20260924 04:44:15.899384 15854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:15.899899 15849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:44:15.900223 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:15.884047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.0.98.62:44145)
0924 04:44:15.884713 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:15.884762 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:15.884859 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:44:15.885721 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:15.885741 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:15.886151 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:15.886470 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:15.887469 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:15.888771 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:15.889967 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:15.890591 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:15.890725 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:15.894028 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:15.894035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:15.894056 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:15.894091 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:15.896400 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:15.896969 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:15.896976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:15.896982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:15.897130 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:15.897547 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:15.897554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:15.897559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:15.898144 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:15.898352 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:15.898842 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 04:44:15.899222 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":197,"threads_started":1}
I20260924 04:44:15.900588 15854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34a89057b31f4a3fa635de52a272bd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } } }
I20260924 04:44:15.900700 15854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: This master's current role is: LEADER
I20260924 04:44:15.900707 15855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a89057b31f4a3fa635de52a272bd34. Latest consensus state: current_term: 1 leader_uuid: "34a89057b31f4a3fa635de52a272bd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } } }
I20260924 04:44:15.900846 15855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: This master's current role is: LEADER
I20260924 04:44:15.902554 15861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:44:15.905771 15861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:44:15.912029 15861 catalog_manager.cc:1347] Generated new cluster ID: fe74404dfcb541aca101d3a63910352d
I20260924 04:44:15.912137 15861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:44:15.941172 15861 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:44:15.942214 15861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:44:15.946471 15861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Generated new TSK 0
I20260924 04:44:15.947194 15861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:44:15.985939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44145
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:44:16.105113 15875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:16.105433 15875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:16.105481 15875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:16.109681 15875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:16.109759 15875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:16.109824 15875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:44:16.115670 15875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:16.116868 15875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:16.118180 15875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:16.128841 15881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:16.128875 15883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:16.128940 15875 server_base.cc:1034] running on GCE node
W20260924 04:44:16.128863 15880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:16.129623 15875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:16.130373 15875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:16.131572 15875 hybrid_clock.cc:648] HybridClock initialized: now 1790225056131553 us; error 34 us; skew 500 ppm
Sep 24 04:44:16 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:16.140483 15875 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:44:16.141515 15875 webserver.cc:460] Webserver started at http://127.0.98.1:43275/ using document root <none> and password file <none>
I20260924 04:44:16.142135 15875 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:16.142204 15875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:16.142393 15875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:44:16.144697 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "35e8b5562b714b9b90ae39bfb9e40be0"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "9d0d6e5a4a9bc857baf32b349e7f073a"
server_key_iv: "9efcf7e6a8be67932462dcdd3011562d"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.145263 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35e8b5562b714b9b90ae39bfb9e40be0"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "9d0d6e5a4a9bc857baf32b349e7f073a"
server_key_iv: "9efcf7e6a8be67932462dcdd3011562d"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.149533 15875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:44:16.152352 15890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.153896 15875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:16.154002 15875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "35e8b5562b714b9b90ae39bfb9e40be0"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "9d0d6e5a4a9bc857baf32b349e7f073a"
server_key_iv: "9efcf7e6a8be67932462dcdd3011562d"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.154101 15875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:16.168257 15875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:16.171819 15875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:16.171975 15875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:16.172518 15875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:16.173408 15875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:16.173455 15875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.173496 15875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:16.173511 15875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.184089 15875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34297
I20260924 04:44:16.184123 16003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34297 every 8 connection(s)
I20260924 04:44:16.185041 15875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:44:16.193305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 15875
I20260924 04:44:16.193483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:44:16.193861 392 external_mini_cluster.cc:1428] Setting key b727447060b1e27d90d9011eb4552d10
Sep 24 04:44:16 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:16.200798 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.186957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57217 (local address 127.0.98.62:44145)
0924 04:44:16.187115 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.187247 (+ 132us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.188063 (+ 816us) server_negotiation.cc:407] Connection header received
0924 04:44:16.189457 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.189461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.189520 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.189604 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:16.190852 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.191495 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.192663 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.192902 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.193021 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.196013 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:16.196020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:16.196026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:16.196061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:16.197858 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.198576 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.198582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.198587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.198707 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.199219 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.199226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.199230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.199501 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:16.199611 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.200303 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.200459 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:44:16.201450 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.187420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:57217)
0924 04:44:16.187890 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.187944 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.189238 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.189789 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.189796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.190183 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:16.190695 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.190709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.191630 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.191639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.192533 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.192542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.193062 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.193066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.193255 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.194031 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:16.194048 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:16.195840 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:16.197998 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:16.198007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:16.198027 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:16.198431 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:16.198828 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:16.198831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:16.198833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:16.199075 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:16.199614 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:16.199619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:16.200007 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.200185 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.200463 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260924 04:44:16.202934 16004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44145
I20260924 04:44:16.203244 16004 heartbeater.cc:464] Registering TS with master...
I20260924 04:44:16.203827 16004 heartbeater.cc:511] Master 127.0.98.62:44145 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:44:16.205489 15813 ts_manager.cc:194] Registered new tserver with Master: 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297)
I20260924 04:44:16.206810 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:44:16.222931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44145
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:44:16.342944 16009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:16.343318 16009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:16.343370 16009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:16.347420 16009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:16.347484 16009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:16.347545 16009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:44:16.353161 16009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:16.354333 16009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:16.355351 16009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:16.365624 16015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:16.365631 16014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:16.365796 16017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:16.365986 16009 server_base.cc:1034] running on GCE node
I20260924 04:44:16.366437 16009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:16.367180 16009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:16.368386 16009 hybrid_clock.cc:648] HybridClock initialized: now 1790225056368368 us; error 33 us; skew 500 ppm
Sep 24 04:44:16 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:16.371414 16009 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:44:16.372356 16009 webserver.cc:460] Webserver started at http://127.0.98.2:33293/ using document root <none> and password file <none>
I20260924 04:44:16.372936 16009 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:16.373026 16009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:16.373291 16009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:44:16.375490 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "72d8c13fd0924ebf965b02adca303ed6"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "a787ce507ab1f488b1ec99fd9e12dcd3"
server_key_iv: "bcdb20467e4dd848407604dfd9bdc2c2"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.376080 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72d8c13fd0924ebf965b02adca303ed6"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "a787ce507ab1f488b1ec99fd9e12dcd3"
server_key_iv: "bcdb20467e4dd848407604dfd9bdc2c2"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.380158 16009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:44:16.382870 16024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.384315 16009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:16.384464 16009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "72d8c13fd0924ebf965b02adca303ed6"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "a787ce507ab1f488b1ec99fd9e12dcd3"
server_key_iv: "bcdb20467e4dd848407604dfd9bdc2c2"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.384590 16009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:16.406847 16009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:16.410535 16009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:16.410741 16009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:16.411425 16009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:16.412262 16009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:16.412333 16009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.412406 16009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:16.412441 16009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.421788 16009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37691
I20260924 04:44:16.421815 16137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37691 every 8 connection(s)
I20260924 04:44:16.422739 16009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:44:16.430562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16009
I20260924 04:44:16.430706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:44:16.431036 392 external_mini_cluster.cc:1428] Setting key 8dade47a509bdea29bc6b3d7b438f6f9
Sep 24 04:44:16 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:16.438762 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.424830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45781 (local address 127.0.98.62:44145)
0924 04:44:16.424989 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.424994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.425972 (+ 978us) server_negotiation.cc:407] Connection header received
0924 04:44:16.427346 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.427349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.427395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.427474 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:16.428813 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.429428 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.430925 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.431286 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.431428 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.434400 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:16.434403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:16.434408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:16.434437 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:16.436068 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.436773 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.436776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.436778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.436862 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.437340 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.437342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.437343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.437523 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:16.437599 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.438467 (+ 868us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.438589 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:44:16.440083 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.425304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:45781)
0924 04:44:16.425803 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.425849 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.427054 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.427686 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.427693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.428115 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:16.428626 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.428640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.429574 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.429577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.430768 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.430781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.431530 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.431535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.431727 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.432466 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:16.432484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:16.434225 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:16.436192 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:16.436199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:16.436216 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:16.436616 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:16.436996 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:16.437002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:16.437007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:16.437229 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:16.437636 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:16.437646 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:16.438125 (+ 479us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.438408 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.438819 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:44:16.441529 16138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44145
I20260924 04:44:16.441763 16138 heartbeater.cc:464] Registering TS with master...
I20260924 04:44:16.442337 16138 heartbeater.cc:511] Master 127.0.98.62:44145 requested a full tablet report, sending...
I20260924 04:44:16.443565 15813 ts_manager.cc:194] Registered new tserver with Master: 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:44:16.444407 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:44:16.462742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44145
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:44:16.580487 16143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:16.580786 16143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:16.580832 16143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:16.585903 16143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:16.585999 16143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:16.586064 16143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:44:16.591919 16143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:16.593062 16143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:16.594239 16143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:16.604928 16148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:16.604986 16151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:16.605100 16143 server_base.cc:1034] running on GCE node
W20260924 04:44:16.604986 16149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:16.605695 16143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:16.606539 16143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:16.607721 16143 hybrid_clock.cc:648] HybridClock initialized: now 1790225056607681 us; error 55 us; skew 500 ppm
Sep 24 04:44:16 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:16.610884 16143 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:44:16.611862 16143 webserver.cc:460] Webserver started at http://127.0.98.3:38399/ using document root <none> and password file <none>
I20260924 04:44:16.612418 16143 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:16.612476 16143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:16.612643 16143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:44:16.614759 16143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "56ef361e784a4e09bc961b68451cae8a"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "033c594f590d8457633c66affca04b6e"
server_key_iv: "170628b82718789767fa8b8bbfeb94ca"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.615357 16143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56ef361e784a4e09bc961b68451cae8a"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "033c594f590d8457633c66affca04b6e"
server_key_iv: "170628b82718789767fa8b8bbfeb94ca"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.619293 16143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:44:16.622010 16158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.623698 16143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:16.623849 16143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "56ef361e784a4e09bc961b68451cae8a"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "033c594f590d8457633c66affca04b6e"
server_key_iv: "170628b82718789767fa8b8bbfeb94ca"
server_key_version: "encryptionkey@0"
I20260924 04:44:16.623988 16143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:16.644254 16143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:16.648136 16143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:16.648383 16143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:16.649058 16143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:16.650002 16143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:16.650054 16143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.650098 16143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:16.650167 16143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:16.660279 16143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32891
I20260924 04:44:16.660318 16271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32891 every 8 connection(s)
I20260924 04:44:16.661310 16143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:44:16.670020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16143
I20260924 04:44:16.670181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:44:16.670499 392 external_mini_cluster.cc:1428] Setting key 291673657327ae7d49164c85d68a6144
Sep 24 04:44:16 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:16.675894 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.663345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44059 (local address 127.0.98.62:44145)
0924 04:44:16.663488 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.663491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.664400 (+ 909us) server_negotiation.cc:407] Connection header received
0924 04:44:16.665772 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.665776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.665836 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.665947 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:16.667238 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.667771 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.669046 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.669275 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.669391 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.672175 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:16.672179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:16.672181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:16.672206 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:16.673769 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.674287 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.674291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.674293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.674374 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.674727 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.674730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.674732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.674921 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:16.675028 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.675605 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.675732 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:44:16.676739 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.663693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:44059)
0924 04:44:16.664231 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.664278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.665510 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.666135 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.666143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.666511 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:16.667019 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.667033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.667896 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.667899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.668908 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.668922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.669431 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.669435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.669618 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.670313 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:16.670334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:16.671989 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:16.673893 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:16.673899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:16.673917 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:16.674183 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:16.674475 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:16.674478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:16.674480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:16.674619 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:16.675038 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:16.675047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:16.675364 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.675550 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.675903 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
I20260924 04:44:16.678054 16272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44145
I20260924 04:44:16.678400 16272 heartbeater.cc:464] Registering TS with master...
I20260924 04:44:16.678920 16272 heartbeater.cc:511] Master 127.0.98.62:44145 requested a full tablet report, sending...
I20260924 04:44:16.680039 15813 ts_manager.cc:194] Registered new tserver with Master: 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
I20260924 04:44:16.680625 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44059
I20260924 04:44:16.684937 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:44:16.695621 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.686907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.0.98.62:44145)
0924 04:44:16.687043 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.687047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.687289 (+ 242us) server_negotiation.cc:407] Connection header received
0924 04:44:16.687332 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.687334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.687373 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.687440 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:16.688093 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.688788 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.689709 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.689935 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.690023 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.691574 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:16.691578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:16.691580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:16.691607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:16.693517 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.694033 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.694036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.694038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.694208 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:16.694519 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:16.694522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:16.694524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:16.694790 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:16.694907 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.695276 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.695433 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:44:16.698724 15813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:44:16.702296 15813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:44:16.717751 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.711518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37691 (local address 127.0.0.1:47924)
0924 04:44:16.711910 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.711966 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.712098 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.712959 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.712970 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.712999 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:16.713190 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.713199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.714736 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.714741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.716006 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.716015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.717143 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.717147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.717267 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.717311 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.717436 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.717575 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":3,"thread_start_us":121,"threads_started":1}
I20260924 04:44:16.717762 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.711735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.0.98.2:37691)
0924 04:44:16.712416 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.712429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.712452 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:44:16.712529 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.712535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.712773 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.713339 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:16.713463 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.714594 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.716151 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.717007 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.717143 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.717306 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.717479 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.717610 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":105,"threads_started":1}
I20260924 04:44:16.717901 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.712393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.98.1:34297)
0924 04:44:16.712792 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.712799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.713020 (+ 221us) server_negotiation.cc:407] Connection header received
0924 04:44:16.713285 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.713292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.713487 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.713647 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:16.713868 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.714662 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.716332 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.717043 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.717149 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.717306 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.717662 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.717754 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":105,"threads_started":1}
I20260924 04:44:16.718152 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.712341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32891 (local address 127.0.0.1:41452)
0924 04:44:16.712941 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.712954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.713053 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.713835 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.713838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.713857 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:16.713949 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.713954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.715514 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.715517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.716593 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.716600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.717698 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.717701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.717799 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.717826 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.717898 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.717984 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260924 04:44:16.718586 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.712730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.0.98.3:32891)
0924 04:44:16.713243 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.713254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.713276 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:44:16.713341 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.713345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.713487 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.713634 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:16.714080 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.715359 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.716714 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.717414 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.717508 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.717640 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.718312 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.718410 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":134,"threads_started":1}
I20260924 04:44:16.718313 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.711804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34297 (local address 127.0.0.1:41196)
0924 04:44:16.712932 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.712945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.713078 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.713631 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.713634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.713651 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:16.713743 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.713749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.714798 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.714802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.716204 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.716214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.717377 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.717380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.717494 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.717519 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.718083 (+ 564us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.718185 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":107,"thread_start_us":95,"threads_started":1}
I20260924 04:44:16.721240 16072 tablet_service.cc:1467] Processing CreateTablet for tablet e58ea71bf28f48618b81c070f4dcef87 (DEFAULT_TABLE table=test-table [id=42c2292ba8634290b7c7912000e0a8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:44:16.721239 16206 tablet_service.cc:1467] Processing CreateTablet for tablet e58ea71bf28f48618b81c070f4dcef87 (DEFAULT_TABLE table=test-table [id=42c2292ba8634290b7c7912000e0a8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:44:16.722291 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58ea71bf28f48618b81c070f4dcef87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:16.722301 16206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58ea71bf28f48618b81c070f4dcef87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:16.723951 15938 tablet_service.cc:1467] Processing CreateTablet for tablet e58ea71bf28f48618b81c070f4dcef87 (DEFAULT_TABLE table=test-table [id=42c2292ba8634290b7c7912000e0a8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:44:16.724994 15938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58ea71bf28f48618b81c070f4dcef87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:16.728855 16289 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Bootstrap starting.
I20260924 04:44:16.729575 16290 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Bootstrap starting.
I20260924 04:44:16.731474 16289 tablet_bootstrap.cc:654] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:16.732348 16290 tablet_bootstrap.cc:654] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:16.732898 16289 log.cc:824] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:16.733179 16290 log.cc:824] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:16.733894 16291 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap starting.
I20260924 04:44:16.734989 16290 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: No bootstrap required, opened a new log
I20260924 04:44:16.735193 16289 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: No bootstrap required, opened a new log
I20260924 04:44:16.735266 16290 ts_tablet_manager.cc:1397] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:44:16.735376 16289 ts_tablet_manager.cc:1397] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:44:16.736232 16291 tablet_bootstrap.cc:654] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:16.737097 16291 log.cc:824] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:16.740259 16289 raft_consensus.cc:348] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.740327 16290 raft_consensus.cc:348] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.740592 16290 raft_consensus.cc:374] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:16.740674 16290 raft_consensus.cc:729] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8c13fd0924ebf965b02adca303ed6, State: Initialized, Role: FOLLOWER
I20260924 04:44:16.740739 16289 raft_consensus.cc:374] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:16.740891 16289 raft_consensus.cc:729] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ef361e784a4e09bc961b68451cae8a, State: Initialized, Role: FOLLOWER
I20260924 04:44:16.741252 16290 consensus_queue.cc:260] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.741653 16289 consensus_queue.cc:260] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.741902 16291 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: No bootstrap required, opened a new log
I20260924 04:44:16.742197 16291 ts_tablet_manager.cc:1397] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 04:44:16.742352 16290 ts_tablet_manager.cc:1428] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:44:16.742741 16138 heartbeater.cc:503] Master 127.0.98.62:44145 was elected leader, sending a full tablet report...
I20260924 04:44:16.742882 16289 ts_tablet_manager.cc:1428] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:44:16.743211 16272 heartbeater.cc:503] Master 127.0.98.62:44145 was elected leader, sending a full tablet report...
I20260924 04:44:16.747103 16291 raft_consensus.cc:348] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.747435 16291 raft_consensus.cc:374] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:16.747546 16291 raft_consensus.cc:729] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8b5562b714b9b90ae39bfb9e40be0, State: Initialized, Role: FOLLOWER
I20260924 04:44:16.748049 16291 consensus_queue.cc:260] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.748807 16004 heartbeater.cc:503] Master 127.0.98.62:44145 was elected leader, sending a full tablet report...
I20260924 04:44:16.749229 16291 ts_tablet_manager.cc:1428] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 04:44:16.912343 16273 tablet.cc:2369] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:44:16.924078 16139 tablet.cc:2369] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:44:16.936463 16005 tablet.cc:2369] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:44:16.974373 16297 raft_consensus.cc:482] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:16.974611 16297 raft_consensus.cc:504] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.975951 16297 leader_election.cc:288] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
I20260924 04:44:16.980727 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.976815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39279 (local address 127.0.98.3:32891)
0924 04:44:16.976962 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.976968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.976988 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:44:16.977057 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.977062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.977128 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.977235 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:16.977719 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.978440 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.979603 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.980022 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.980147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.980217 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.980500 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.980576 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:44:16.980823 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.976647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32891 (local address 127.0.98.1:39279)
0924 04:44:16.976893 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.976905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.976981 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.977386 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.977392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.977418 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:16.977567 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.977576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.978551 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.978557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.979470 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.979480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.980196 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.980201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.980336 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.980380 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.980472 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.980576 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260924 04:44:16.981377 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.976197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37691 (local address 127.0.98.1:43169)
0924 04:44:16.976516 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:16.976533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:16.976667 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:16.977255 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:16.977258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:16.977272 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:16.977408 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.977415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.978429 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.978434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.979981 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:16.979996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.980893 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.980898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:16.981033 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.981069 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:44:16.981148 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:44:16.981238 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":135,"threads_started":1}
I20260924 04:44:16.981627 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:16.976318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43169 (local address 127.0.98.2:37691)
0924 04:44:16.976490 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:16.976497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:16.977056 (+ 559us) server_negotiation.cc:407] Connection header received
0924 04:44:16.977107 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:16.977109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:16.977149 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:16.977247 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:16.977520 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.978304 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.980127 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:16.980656 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:16.980763 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:16.980829 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:16.981405 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 04:44:16.981513 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:44:16.981747 16226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ea71bf28f48618b81c070f4dcef87" candidate_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ef361e784a4e09bc961b68451cae8a" is_pre_election: true
I20260924 04:44:16.982139 16226 raft_consensus.cc:2454] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e8b5562b714b9b90ae39bfb9e40be0 in term 0.
I20260924 04:44:16.982350 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ea71bf28f48618b81c070f4dcef87" candidate_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d8c13fd0924ebf965b02adca303ed6" is_pre_election: true
I20260924 04:44:16.982690 16092 raft_consensus.cc:2454] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e8b5562b714b9b90ae39bfb9e40be0 in term 0.
I20260924 04:44:16.982738 15891 leader_election.cc:302] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0, 56ef361e784a4e09bc961b68451cae8a; no voters:
I20260924 04:44:16.983016 16297 raft_consensus.cc:2781] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:44:16.983093 16297 raft_consensus.cc:482] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:44:16.983170 16297 raft_consensus.cc:3037] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:16.984186 16297 raft_consensus.cc:504] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.984532 16297 leader_election.cc:288] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 1 election: Requested vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
I20260924 04:44:16.984927 16226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ea71bf28f48618b81c070f4dcef87" candidate_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ef361e784a4e09bc961b68451cae8a"
I20260924 04:44:16.984913 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ea71bf28f48618b81c070f4dcef87" candidate_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d8c13fd0924ebf965b02adca303ed6"
I20260924 04:44:16.985083 16226 raft_consensus.cc:3037] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:16.985087 16092 raft_consensus.cc:3037] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:16.986338 16226 raft_consensus.cc:2454] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e8b5562b714b9b90ae39bfb9e40be0 in term 1.
I20260924 04:44:16.986685 15891 leader_election.cc:302] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0, 56ef361e784a4e09bc961b68451cae8a; no voters:
I20260924 04:44:16.986778 16092 raft_consensus.cc:2454] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e8b5562b714b9b90ae39bfb9e40be0 in term 1.
I20260924 04:44:16.986887 16297 raft_consensus.cc:2781] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:44:16.987098 16297 raft_consensus.cc:686] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 LEADER]: Becoming Leader. State: Replica: 35e8b5562b714b9b90ae39bfb9e40be0, State: Running, Role: LEADER
I20260924 04:44:16.987486 16297 consensus_queue.cc:237] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:16.991325 15812 catalog_manager.cc:5495] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } health_report { overall_health: HEALTHY } } }
I20260924 04:44:17.012030 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:17.008303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.0.98.1:34297)
0924 04:44:17.008464 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:17.008467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:17.008481 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:44:17.008622 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:17.008625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:17.008685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:17.008780 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:44:17.009026 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:17.009587 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:17.010484 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:17.010700 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:17.010810 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:17.011090 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:44:17.011602 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:44:17.011697 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:17.011862 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:44:17.011918 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 04:44:17.018635 16226 raft_consensus.cc:1261] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Refusing update from remote peer 35e8b5562b714b9b90ae39bfb9e40be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:44:17.018663 16092 raft_consensus.cc:1261] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Refusing update from remote peer 35e8b5562b714b9b90ae39bfb9e40be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:44:17.019486 16300 consensus_queue.cc:1035] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:44:17.019724 16297 consensus_queue.cc:1035] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:44:17.037678 15813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:44:17.037956 15813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:44:17.040683 15813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Sending DeleteTablet for 3 replicas of tablet e58ea71bf28f48618b81c070f4dcef87
I20260924 04:44:17.041903 16072 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ea71bf28f48618b81c070f4dcef87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47924
I20260924 04:44:17.041919 16206 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ea71bf28f48618b81c070f4dcef87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41452
I20260924 04:44:17.042511 16313 tablet_replica.cc:330] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: stopping tablet replica
I20260924 04:44:17.042481 15938 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ea71bf28f48618b81c070f4dcef87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41196
I20260924 04:44:17.042884 16313 raft_consensus.cc:2229] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:44:17.042965 16315 tablet_replica.cc:330] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: stopping tablet replica
I20260924 04:44:17.043040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 15875
I20260924 04:44:17.043375 16313 raft_consensus.cc:2258] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:44:17.043368 16315 raft_consensus.cc:2229] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:44:17.043936 16314 tablet_replica.cc:330] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: stopping tablet replica
I20260924 04:44:17.044232 16314 raft_consensus.cc:2229] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:44:17.044543 16314 raft_consensus.cc:2258] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:44:17.045015 16313 ts_tablet_manager.cc:1905] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:44:17.046326 16314 ts_tablet_manager.cc:1905] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:44:17.048504 16314 ts_tablet_manager.cc:1918] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:44:17.048630 16314 log.cc:1196] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/e58ea71bf28f48618b81c070f4dcef87
I20260924 04:44:17.048815 16313 ts_tablet_manager.cc:1918] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:44:17.048924 16313 log.cc:1196] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/e58ea71bf28f48618b81c070f4dcef87
I20260924 04:44:17.048987 16314 ts_tablet_manager.cc:1939] T e58ea71bf28f48618b81c070f4dcef87 P 56ef361e784a4e09bc961b68451cae8a: Deleting consensus metadata
I20260924 04:44:17.049307 16313 ts_tablet_manager.cc:1939] T e58ea71bf28f48618b81c070f4dcef87 P 72d8c13fd0924ebf965b02adca303ed6: Deleting consensus metadata
I20260924 04:44:17.050310 15797 catalog_manager.cc:4841] TS 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): tablet e58ea71bf28f48618b81c070f4dcef87 (table test-table [id=42c2292ba8634290b7c7912000e0a8c7]) successfully deleted
I20260924 04:44:17.050575 15797 catalog_manager.cc:4841] TS 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): tablet e58ea71bf28f48618b81c070f4dcef87 (table test-table [id=42c2292ba8634290b7c7912000e0a8c7]) successfully deleted
W20260924 04:44:17.051800 15800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34297 (error 108)
I20260924 04:44:17.052407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16009
I20260924 04:44:17.053485 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:17.053130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34297 (local address 127.0.0.1:41216)
0924 04:44:17.053289 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:17.053374 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 04:44:17.053681 15800 catalog_manager.cc:4584] TS 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e58ea71bf28f48618b81c070f4dcef87: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34297: connect: Connection refused (error 111)
I20260924 04:44:17.058804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16143
I20260924 04:44:17.064754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 15781
I20260924 04:44:37.071589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44145
--webserver_interface=127.0.98.62
--webserver_port=37263
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:44:37.185839 16317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:37.186120 16317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:37.186160 16317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:37.190402 16317 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:44:37.190459 16317 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:44:37.190476 16317 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:44:37.190490 16317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:44:37.190514 16317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:37.190532 16317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:37.190579 16317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:44:37.190611 16317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:44:37.196025 16317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:37.197147 16317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:37.198210 16317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:37.212090 16324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.212247 16326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.212316 16323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:37.212569 16317 server_base.cc:1034] running on GCE node
I20260924 04:44:37.213286 16317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:37.214648 16317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:37.215852 16317 hybrid_clock.cc:648] HybridClock initialized: now 1790225077215759 us; error 110 us; skew 500 ppm
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:37.225958 16317 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:44:37.227072 16317 webserver.cc:460] Webserver started at http://127.0.98.62:37263/ using document root <none> and password file <none>
I20260924 04:44:37.227830 16317 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:37.227952 16317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:37.232604 16317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:44:37.235309 16333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.236685 16317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:37.236836 16317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "34a89057b31f4a3fa635de52a272bd34"
format_stamp: "Formatted at 2026-09-24 04:44:15 on dist-test-slave-npg2"
server_key: "973530e5a9d5e9a28c0d2be6b8f38f0a"
server_key_iv: "cff2f7ca017289c386a6c94278ce3f89"
server_key_version: "encryptionkey@0"
I20260924 04:44:37.237468 16317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:37.263727 16317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:37.267457 16317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:37.267700 16317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:37.275019 16317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44145
I20260924 04:44:37.275399 16385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44145 every 8 connection(s)
I20260924 04:44:37.276225 16317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:44:37.278791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16317
I20260924 04:44:37.279290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34297
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44145
--webserver_port=43275
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:44:37.281057 16386 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:44:37.283373 16386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap starting.
I20260924 04:44:37.308142 16386 log.cc:824] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:37.319800 16386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:44:37.320262 16386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap complete.
I20260924 04:44:37.326043 16386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:37.326582 16386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a89057b31f4a3fa635de52a272bd34, State: Initialized, Role: FOLLOWER
I20260924 04:44:37.327175 16386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:37.327328 16386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:44:37.327394 16386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:44:37.327492 16386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:44:37.330343 16386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:37.330749 16386 leader_election.cc:302] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34a89057b31f4a3fa635de52a272bd34; no voters:
I20260924 04:44:37.331189 16386 leader_election.cc:288] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:44:37.331388 16390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:44:37.331777 16390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 LEADER]: Becoming Leader. State: Replica: 34a89057b31f4a3fa635de52a272bd34, State: Running, Role: LEADER
I20260924 04:44:37.332178 16390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:37.332532 16386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:44:37.333838 16392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a89057b31f4a3fa635de52a272bd34. Latest consensus state: current_term: 2 leader_uuid: "34a89057b31f4a3fa635de52a272bd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } } }
I20260924 04:44:37.333971 16392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: This master's current role is: LEADER
I20260924 04:44:37.333822 16391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34a89057b31f4a3fa635de52a272bd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } } }
I20260924 04:44:37.334106 16391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: This master's current role is: LEADER
I20260924 04:44:37.334549 16400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:44:37.339751 16400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:44:37.340466 16400 catalog_manager.cc:1259] Loaded cluster ID: fe74404dfcb541aca101d3a63910352d
I20260924 04:44:37.340543 16400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:44:37.343200 16400 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:44:37.343946 16400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Loaded TSK: 0
I20260924 04:44:37.364024 16400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Generated new TSK 1
I20260924 04:44:37.364326 16400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:44:37.415563 16388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:37.415920 16388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:37.416026 16388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:37.420358 16388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:37.420454 16388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:37.420549 16388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:44:37.426288 16388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43275
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:37.427495 16388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:37.428545 16388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:37.438654 16414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.438714 16416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.438624 16413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:37.439350 16388 server_base.cc:1034] running on GCE node
I20260924 04:44:37.439827 16388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:37.440577 16388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:37.441761 16388 hybrid_clock.cc:648] HybridClock initialized: now 1790225077441749 us; error 31 us; skew 500 ppm
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:37.444729 16388 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:44:37.445799 16388 webserver.cc:460] Webserver started at http://127.0.98.1:43275/ using document root <none> and password file <none>
I20260924 04:44:37.446373 16388 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:37.446448 16388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:37.450721 16388 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:44:37.453135 16423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.454479 16388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:44:37.454581 16388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "35e8b5562b714b9b90ae39bfb9e40be0"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "9d0d6e5a4a9bc857baf32b349e7f073a"
server_key_iv: "9efcf7e6a8be67932462dcdd3011562d"
server_key_version: "encryptionkey@0"
I20260924 04:44:37.455245 16388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:37.492671 16388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:37.496397 16388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:37.496680 16388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:37.497341 16388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:37.498508 16430 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:44:37.503252 16388 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:44:37.503325 16388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 04:44:37.503407 16388 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:44:37.505228 16388 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:44:37.505278 16388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:44:37.505596 16430 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap starting.
I20260924 04:44:37.516227 16388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34297
I20260924 04:44:37.516281 16537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34297 every 8 connection(s)
I20260924 04:44:37.517263 16388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:44:37.524616 16430 log.cc:824] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:37.527640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16388
I20260924 04:44:37.528231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37691
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44145
--webserver_port=33293
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:37.533856 16430 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:44:37.534341 16430 tablet_bootstrap.cc:492] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap complete.
I20260924 04:44:37.535677 16430 ts_tablet_manager.cc:1397] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.003s
I20260924 04:44:37.536849 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.519384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48595 (local address 127.0.98.62:44145)
0924 04:44:37.520144 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.520164 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.520214 (+ 50us) server_negotiation.cc:407] Connection header received
0924 04:44:37.521305 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.521322 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.521756 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.522195 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:37.523427 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.524911 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.526469 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.527179 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.527379 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.530440 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:37.530446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:37.530464 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:37.530499 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:37.532756 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.533455 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.533461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.533467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.533626 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.534087 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.534092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.534095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.534685 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:37.534825 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.535601 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.535963 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":118,"threads_started":1}
I20260924 04:44:37.536952 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.519403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:48595)
0924 04:44:37.519939 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.519996 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.520829 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.522296 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.522303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.522687 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:37.523243 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.523260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.525217 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.525223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.526334 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.526345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.527361 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.527365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.527585 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.528315 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:37.528339 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:37.530222 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:37.532904 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.532910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.532923 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.533304 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.533753 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.533760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.533762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.533966 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.534820 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:37.534829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:37.535220 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.535455 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.535872 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260924 04:44:37.538694 16430 raft_consensus.cc:348] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.539206 16430 raft_consensus.cc:729] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8b5562b714b9b90ae39bfb9e40be0, State: Initialized, Role: FOLLOWER
I20260924 04:44:37.539736 16430 consensus_queue.cc:260] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.539829 16538 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44145
I20260924 04:44:37.540084 16538 heartbeater.cc:464] Registering TS with master...
I20260924 04:44:37.540756 16538 heartbeater.cc:511] Master 127.0.98.62:44145 requested a full tablet report, sending...
I20260924 04:44:37.540817 16430 ts_tablet_manager.cc:1428] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:44:37.543445 16350 ts_manager.cc:194] Registered new tserver with Master: 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297)
I20260924 04:44:37.546936 16350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48595
I20260924 04:44:37.549836 16538 heartbeater.cc:503] Master 127.0.98.62:44145 was elected leader, sending a full tablet report...
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:44:37.556857 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.546256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34297 (local address 127.0.0.1:55210)
0924 04:44:37.547088 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.547152 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.547289 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.547876 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.547883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.548272 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:37.548381 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.548387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.549347 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.549352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.550055 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.550064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.550719 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.550737 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.550863 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.551537 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:37.551551 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:37.553011 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:37.554932 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.554942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.554948 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.555263 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.555564 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.555568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.555570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.555659 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.556118 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:37.556124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:37.556324 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.556461 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.556668 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":108,"threads_started":1}
I20260924 04:44:37.556913 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.546398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.98.1:34297)
0924 04:44:37.546880 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.546893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.547248 (+ 355us) server_negotiation.cc:407] Connection header received
0924 04:44:37.547436 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.547442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.547654 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.547793 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:37.548512 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.549229 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.550178 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.550610 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.550734 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.553164 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:37.553172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:37.553183 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:37.553206 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:37.554810 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.555386 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.555391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.555392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.555468 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.555756 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.555760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.555761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.556001 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:37.556143 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.556556 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.556769 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":135,"threads_started":1}
I20260924 04:44:37.557885 16472 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ea71bf28f48618b81c070f4dcef87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55210
I20260924 04:44:37.558789 16547 tablet_replica.cc:330] stopping tablet replica
I20260924 04:44:37.559108 16547 raft_consensus.cc:2229] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:44:37.559433 16547 raft_consensus.cc:2258] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:44:37.560544 16547 ts_tablet_manager.cc:1905] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:44:37.564491 16547 ts_tablet_manager.cc:1918] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:44:37.564586 16547 log.cc:1196] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal/wals/e58ea71bf28f48618b81c070f4dcef87
I20260924 04:44:37.564855 16547 ts_tablet_manager.cc:1939] T e58ea71bf28f48618b81c070f4dcef87 P 35e8b5562b714b9b90ae39bfb9e40be0: Deleting consensus metadata
I20260924 04:44:37.566666 16337 catalog_manager.cc:4841] TS 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297): tablet e58ea71bf28f48618b81c070f4dcef87 (table test-table [id=42c2292ba8634290b7c7912000e0a8c7]) successfully deleted
W20260924 04:44:37.658509 16543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:37.658854 16543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:37.658931 16543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:37.663576 16543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:37.663642 16543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:37.663729 16543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:44:37.669269 16543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33293
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:37.670372 16543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:37.671520 16543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:37.681298 16553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.681313 16552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.681298 16555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:37.681581 16543 server_base.cc:1034] running on GCE node
I20260924 04:44:37.682068 16543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:37.682766 16543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:37.683982 16543 hybrid_clock.cc:648] HybridClock initialized: now 1790225077683909 us; error 80 us; skew 500 ppm
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:37.686929 16543 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:44:37.687899 16543 webserver.cc:460] Webserver started at http://127.0.98.2:33293/ using document root <none> and password file <none>
I20260924 04:44:37.688495 16543 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:37.688612 16543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:37.692739 16543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:44:37.695254 16562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.696677 16543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:44:37.696837 16543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "72d8c13fd0924ebf965b02adca303ed6"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "a787ce507ab1f488b1ec99fd9e12dcd3"
server_key_iv: "bcdb20467e4dd848407604dfd9bdc2c2"
server_key_version: "encryptionkey@0"
I20260924 04:44:37.697495 16543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:37.707686 16543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:37.710762 16543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:37.710961 16543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:37.711601 16543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:37.712402 16543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:37.712469 16543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.712536 16543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:37.712584 16543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.721650 16543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37691
I20260924 04:44:37.721700 16675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37691 every 8 connection(s)
I20260924 04:44:37.722708 16543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:44:37.726732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16543
I20260924 04:44:37.727239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32891
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44145
--webserver_port=38399
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:37.738534 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.724733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60587 (local address 127.0.98.62:44145)
0924 04:44:37.724897 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.724901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.725681 (+ 780us) server_negotiation.cc:407] Connection header received
0924 04:44:37.727165 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.727169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.727250 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.727347 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:37.728708 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.729256 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.731192 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.731539 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.731638 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.734758 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:37.734762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:37.734765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:37.734800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:37.736381 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.736926 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.736930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.736932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.737013 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.737372 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.737376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.737378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.737571 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:37.737683 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.738209 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.738344 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:44:37.739576 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.725082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:60587)
0924 04:44:37.725518 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.725570 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.726900 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.727546 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.727557 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.727932 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:37.728472 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.728494 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.729393 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.729398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.730961 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.730977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.731715 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.731721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.731967 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.732701 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:37.732718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:37.734585 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:37.736506 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.736512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.736529 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.736820 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.737124 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.737127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.737129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.737276 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.737694 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:37.737702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:37.737965 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.738170 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.738558 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260924 04:44:37.740829 16676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44145
I20260924 04:44:37.741080 16676 heartbeater.cc:464] Registering TS with master...
I20260924 04:44:37.741667 16676 heartbeater.cc:511] Master 127.0.98.62:44145 requested a full tablet report, sending...
I20260924 04:44:37.742736 16350 ts_manager.cc:194] Registered new tserver with Master: 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691)
I20260924 04:44:37.743367 16350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60587
W20260924 04:44:37.843473 16679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:37.843786 16679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:37.843876 16679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:37.848013 16679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:37.848215 16679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:37.848304 16679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:44:37.853819 16679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38399
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:37.854970 16679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:37.856123 16679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:37.865923 16684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.865931 16687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:37.865906 16685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:37.866269 16679 server_base.cc:1034] running on GCE node
I20260924 04:44:37.866658 16679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:37.867362 16679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:37.868561 16679 hybrid_clock.cc:648] HybridClock initialized: now 1790225077868539 us; error 40 us; skew 500 ppm
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:37.871673 16679 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:44:37.872720 16679 webserver.cc:460] Webserver started at http://127.0.98.3:38399/ using document root <none> and password file <none>
I20260924 04:44:37.873391 16679 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:37.873503 16679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:37.877966 16679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:44:37.880316 16694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.881767 16679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:37.881933 16679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "56ef361e784a4e09bc961b68451cae8a"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "033c594f590d8457633c66affca04b6e"
server_key_iv: "170628b82718789767fa8b8bbfeb94ca"
server_key_version: "encryptionkey@0"
I20260924 04:44:37.882531 16679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:37.895956 16679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:37.899861 16679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:37.900064 16679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:37.900700 16679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:37.901582 16679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:37.901665 16679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.901722 16679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:37.901751 16679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:37.911414 16679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32891
I20260924 04:44:37.911448 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32891 every 8 connection(s)
I20260924 04:44:37.912457 16679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:44:37.914312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16679
I20260924 04:44:37.924124 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.914996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.0.98.62:44145)
0924 04:44:37.915270 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.915275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.915424 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:44:37.915561 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.915563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.915635 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.915719 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:37.916709 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.917281 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.918604 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.918839 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.918931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.920473 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:37.920477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:37.920480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:37.920506 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:37.922279 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.922780 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.922786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.922788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.922886 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.923238 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.923240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.923242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.923436 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:37.923719 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.923808 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.923936 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":66,"threads_started":1}
Sep 24 04:44:37 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:44:37.929157 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.914475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38105 (local address 127.0.98.62:44145)
0924 04:44:37.914658 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.914662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.916419 (+ 1757us) server_negotiation.cc:407] Connection header received
0924 04:44:37.917973 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.917976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.918014 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.918082 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:37.920037 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.920769 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.922232 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.922540 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.922656 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.925451 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:37.925454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:37.925455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:37.925477 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:37.926926 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.927638 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.927641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.927642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.927717 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.928084 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.928087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.928089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.928289 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:37.928373 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.928918 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.929019 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 04:44:37.930073 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.914947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:38105)
0924 04:44:37.916241 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.916307 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.917743 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.918270 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.918279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.918955 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:44:37.919774 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.919796 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.920921 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.920925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.922093 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.922103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.922695 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.922699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.922874 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.923581 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:44:37.923598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:44:37.925283 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:44:37.927031 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.927036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.927193 (+ 157us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.927484 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.927802 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:44:37.927805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:44:37.927806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:44:37.927953 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:44:37.928386 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:44:37.928392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:44:37.928652 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.928856 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.929133 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":184,"threads_started":1}
I20260924 04:44:37.931347 16808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44145
I20260924 04:44:37.931643 16808 heartbeater.cc:464] Registering TS with master...
I20260924 04:44:37.932183 16808 heartbeater.cc:511] Master 127.0.98.62:44145 requested a full tablet report, sending...
I20260924 04:44:37.933393 16350 ts_manager.cc:194] Registered new tserver with Master: 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
I20260924 04:44:37.933992 16350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38105
I20260924 04:44:37.937984 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:44:37.942301 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.938634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.0.98.62:44145)
0924 04:44:37.938798 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.938802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.939020 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:44:37.939251 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.939254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.939357 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.939447 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:44:37.939664 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.940161 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.941132 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.941382 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.941461 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.941713 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:44:37.941862 (+ 149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 04:44:37.942161 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55694: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 04:44:37.942382 16540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55694: token expired
W20260924 04:44:37.942706 16275 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:44145)
I20260924 04:44:37.950040 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.943076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.98.62:44145)
0924 04:44:37.943258 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.943262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.943276 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:44:37.943351 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.943354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.943400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.943459 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:44:37.944173 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.944682 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.945487 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.945679 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.945759 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.946989 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:44:37.946992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:44:37.946994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:44:37.947016 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:44:37.948577 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.948982 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.948984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.948986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.949057 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:44:37.949306 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:44:37.949308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:44:37.949309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:44:37.949467 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:44:37.949563 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.949767 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.949852 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 04:44:37.952591 16350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:44:37.956144 16350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:44:37.963591 16472 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2db53fea97428ab4440ba1f5d1c57d (DEFAULT_TABLE table=test-table [id=31969cfb6da7497ca8fec55b10496533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:44:37.964468 16472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2db53fea97428ab4440ba1f5d1c57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:37.967839 16430 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap starting.
I20260924 04:44:37.968631 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.962721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37691 (local address 127.0.0.1:52708)
0924 04:44:37.962850 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.962868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.962946 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.964164 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.964167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.964194 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:37.964282 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.964287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.965626 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.965629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.966868 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.966878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.968130 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.968133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.968257 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.968310 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.968400 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.968490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":33}
I20260924 04:44:37.968724 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.963294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.0.98.2:37691)
0924 04:44:37.963623 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.963635 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.963658 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:44:37.963740 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.963747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.963980 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.964414 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:37.964607 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.965481 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.967011 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.967980 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.968098 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.968258 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.968452 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.968560 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":155,"threads_started":1}
I20260924 04:44:37.968837 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.962989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32891 (local address 127.0.0.1:37662)
0924 04:44:37.963270 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.963284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.963370 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.964659 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.964662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.964679 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:37.964761 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.964766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.966514 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.966517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.967482 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.967491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.968465 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.968468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.968545 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.968565 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.968610 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.968692 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":24,"thread_start_us":138,"threads_started":1}
I20260924 04:44:37.968899 16430 tablet_bootstrap.cc:654] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:37.969444 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.963659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.0.98.3:32891)
0924 04:44:37.964066 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.964079 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.964105 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:44:37.964190 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.964196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.964394 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.964566 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:37.964884 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.966381 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.967607 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.968336 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.968973 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.969130 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.969188 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.969275 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":118,"threads_started":1}
I20260924 04:44:37.970726 16430 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: No bootstrap required, opened a new log
I20260924 04:44:37.970818 16430 ts_tablet_manager.cc:1397] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:44:37.971323 16430 raft_consensus.cc:348] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.971455 16430 raft_consensus.cc:374] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:37.971493 16430 raft_consensus.cc:729] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8b5562b714b9b90ae39bfb9e40be0, State: Initialized, Role: FOLLOWER
I20260924 04:44:37.971649 16430 consensus_queue.cc:260] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.972373 16430 ts_tablet_manager.cc:1428] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:44:37.972620 16742 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2db53fea97428ab4440ba1f5d1c57d (DEFAULT_TABLE table=test-table [id=31969cfb6da7497ca8fec55b10496533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:44:37.972638 16610 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2db53fea97428ab4440ba1f5d1c57d (DEFAULT_TABLE table=test-table [id=31969cfb6da7497ca8fec55b10496533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:44:37.973742 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2db53fea97428ab4440ba1f5d1c57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:37.973824 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2db53fea97428ab4440ba1f5d1c57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:44:37.980800 16821 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Bootstrap starting.
I20260924 04:44:37.981314 16822 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Bootstrap starting.
I20260924 04:44:37.983487 16821 tablet_bootstrap.cc:654] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:37.983863 16822 tablet_bootstrap.cc:654] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Neither blocks nor log segments found. Creating new log.
I20260924 04:44:37.984572 16822 log.cc:824] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:37.984573 16821 log.cc:824] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:37.986461 16821 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: No bootstrap required, opened a new log
I20260924 04:44:37.986522 16822 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: No bootstrap required, opened a new log
I20260924 04:44:37.986723 16821 ts_tablet_manager.cc:1397] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:44:37.986728 16822 ts_tablet_manager.cc:1397] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:44:37.989861 16821 raft_consensus.cc:348] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.989861 16822 raft_consensus.cc:348] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.990094 16821 raft_consensus.cc:374] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:37.990123 16822 raft_consensus.cc:374] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:44:37.990155 16821 raft_consensus.cc:729] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ef361e784a4e09bc961b68451cae8a, State: Initialized, Role: FOLLOWER
I20260924 04:44:37.990217 16822 raft_consensus.cc:729] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8c13fd0924ebf965b02adca303ed6, State: Initialized, Role: FOLLOWER
I20260924 04:44:37.990639 16821 consensus_queue.cc:260] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.990646 16822 consensus_queue.cc:260] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.991364 16676 heartbeater.cc:503] Master 127.0.98.62:44145 was elected leader, sending a full tablet report...
I20260924 04:44:37.991598 16808 heartbeater.cc:503] Master 127.0.98.62:44145 was elected leader, sending a full tablet report...
I20260924 04:44:37.991667 16821 ts_tablet_manager.cc:1428] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:44:37.991685 16822 ts_tablet_manager.cc:1428] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:44:37.992667 16827 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:44:37.992832 16827 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:37.994441 16827 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297)
I20260924 04:44:37.998613 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.994434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59903 (local address 127.0.98.1:34297)
0924 04:44:37.994679 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.994684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.994700 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:44:37.994757 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.994759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.994846 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.994949 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:37.995295 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.996172 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.997626 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.998076 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.998220 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.998393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.998430 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.998500 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260924 04:44:37.998613 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.994344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34297 (local address 127.0.98.3:59903)
0924 04:44:37.994504 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.994522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.994664 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.994981 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.994983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.994999 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:37.995091 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.995096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.996291 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.996294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.997508 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.997516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.998217 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.998221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.998320 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.998341 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.998406 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.998480 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":2}
I20260924 04:44:37.999466 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.995418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37691 (local address 127.0.98.3:49279)
0924 04:44:37.995840 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:37.995856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:37.995953 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:37.996196 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:44:37.996198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:44:37.996215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:44:37.996345 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.996354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.997395 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.997398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.998167 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:44:37.998175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.999035 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.999039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:44:37.999170 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.999196 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:44:37.999280 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:44:37.999350 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":105,"threads_started":1}
I20260924 04:44:37.999529 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:37.995498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49279 (local address 127.0.98.2:37691)
0924 04:44:37.995602 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:37.995607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:37.995939 (+ 332us) server_negotiation.cc:407] Connection header received
0924 04:44:37.996034 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:37.996037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:37.996077 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:37.996161 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:44:37.996463 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.997251 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.998303 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:37.998849 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:37.998953 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:37.999010 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:37.999324 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 04:44:37.999419 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:44:37.999594 16492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2db53fea97428ab4440ba1f5d1c57d" candidate_uuid: "56ef361e784a4e09bc961b68451cae8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" is_pre_election: true
I20260924 04:44:37.999909 16492 raft_consensus.cc:2454] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ef361e784a4e09bc961b68451cae8a in term 0.
I20260924 04:44:38.000604 16698 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0, 56ef361e784a4e09bc961b68451cae8a; no voters:
I20260924 04:44:38.000658 16630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2db53fea97428ab4440ba1f5d1c57d" candidate_uuid: "56ef361e784a4e09bc961b68451cae8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d8c13fd0924ebf965b02adca303ed6" is_pre_election: true
I20260924 04:44:38.000927 16827 raft_consensus.cc:2781] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:44:38.000934 16630 raft_consensus.cc:2454] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ef361e784a4e09bc961b68451cae8a in term 0.
I20260924 04:44:38.001015 16827 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:44:38.001049 16827 raft_consensus.cc:3037] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:38.002519 16827 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:38.002931 16827 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [CANDIDATE]: Term 1 election: Requested vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297)
I20260924 04:44:38.003340 16630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2db53fea97428ab4440ba1f5d1c57d" candidate_uuid: "56ef361e784a4e09bc961b68451cae8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d8c13fd0924ebf965b02adca303ed6"
I20260924 04:44:38.003403 16492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2db53fea97428ab4440ba1f5d1c57d" candidate_uuid: "56ef361e784a4e09bc961b68451cae8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e8b5562b714b9b90ae39bfb9e40be0"
I20260924 04:44:38.003487 16630 raft_consensus.cc:3037] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:38.003556 16492 raft_consensus.cc:3037] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:44:38.004558 16630 raft_consensus.cc:2454] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ef361e784a4e09bc961b68451cae8a in term 1.
I20260924 04:44:38.004691 16492 raft_consensus.cc:2454] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ef361e784a4e09bc961b68451cae8a in term 1.
I20260924 04:44:38.004899 16695 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ef361e784a4e09bc961b68451cae8a, 72d8c13fd0924ebf965b02adca303ed6; no voters:
I20260924 04:44:38.005064 16827 raft_consensus.cc:2781] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:44:38.005177 16827 raft_consensus.cc:686] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 1 LEADER]: Becoming Leader. State: Replica: 56ef361e784a4e09bc961b68451cae8a, State: Running, Role: LEADER
I20260924 04:44:38.005497 16827 consensus_queue.cc:237] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:38.009265 16349 catalog_manager.cc:5495] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 56ef361e784a4e09bc961b68451cae8a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "56ef361e784a4e09bc961b68451cae8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } health_report { overall_health: UNKNOWN } } }
W20260924 04:44:38.018584 16539 tablet.cc:2369] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:44:38.030638 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:38.027084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.98.3:32891)
0924 04:44:38.027282 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 04:44:38.027288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:44:38.027302 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:44:38.027410 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:44:38.027412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:44:38.027468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:44:38.027586 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:44:38.027830 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:38.028338 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:38.029166 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:44:38.029391 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:44:38.029501 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:44:38.029813 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:44:38.030239 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:44:38.030374 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:44:38.030434 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 04:44:38.030493 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 04:44:38.037477 16630 raft_consensus.cc:1261] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Refusing update from remote peer 56ef361e784a4e09bc961b68451cae8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:44:38.037475 16492 raft_consensus.cc:1261] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Refusing update from remote peer 56ef361e784a4e09bc961b68451cae8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:44:38.038265 16827 consensus_queue.cc:1035] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:44:38.038546 16826 consensus_queue.cc:1035] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:44:38.046892 16832 mvcc.cc:205] Tried to move back new op lower bound from 7332761919633338368 to 7332761919513530368. Current Snapshot: MvccSnapshot[applied={T|T < 7332761919633338368}]
I20260924 04:44:38.047025 16833 mvcc.cc:205] Tried to move back new op lower bound from 7332761919633338368 to 7332761919513530368. Current Snapshot: MvccSnapshot[applied={T|T < 7332761919633338368}]
I20260924 04:44:38.053821 16350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:44:38.054028 16350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:44:38.056726 16350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Sending DeleteTablet for 3 replicas of tablet 2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:44:38.057570 16472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2db53fea97428ab4440ba1f5d1c57d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55210
I20260924 04:44:38.057650 16610 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2db53fea97428ab4440ba1f5d1c57d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52708
I20260924 04:44:38.057808 16547 tablet_replica.cc:330] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: stopping tablet replica
I20260924 04:44:38.057952 16547 raft_consensus.cc:2229] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:44:38.058095 16547 raft_consensus.cc:2258] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:44:38.058288 16840 tablet_replica.cc:330] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: stopping tablet replica
I20260924 04:44:38.058305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16388
I20260924 04:44:38.058645 16840 raft_consensus.cc:2229] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:44:38.058990 16840 raft_consensus.cc:2258] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:44:38.060730 16840 ts_tablet_manager.cc:1905] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:44:38.063505 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2db53fea97428ab4440ba1f5d1c57d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:44:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37662
I20260924 04:44:38.064113 16841 tablet_replica.cc:330] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: stopping tablet replica
I20260924 04:44:38.064309 16840 ts_tablet_manager.cc:1918] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:44:38.064419 16840 log.cc:1196] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:44:38.064589 16841 raft_consensus.cc:2229] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:44:38.064771 16840 ts_tablet_manager.cc:1939] T 2a2db53fea97428ab4440ba1f5d1c57d P 72d8c13fd0924ebf965b02adca303ed6: Deleting consensus metadata
I20260924 04:44:38.065140 16841 raft_consensus.cc:2258] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:44:38.066171 16334 catalog_manager.cc:4841] TS 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): tablet 2a2db53fea97428ab4440ba1f5d1c57d (table test-table [id=31969cfb6da7497ca8fec55b10496533]) successfully deleted
I20260924 04:44:38.066910 16841 ts_tablet_manager.cc:1905] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:44:38.067525 16337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34297 (error 108)
W20260924 04:44:38.068908 16817 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:44:38.068549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34297 (local address 127.0.0.1:55226)
0924 04:44:38.068699 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:38.068717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:38.068798 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34297: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 04:44:38.069095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16543
I20260924 04:44:38.069648 16841 ts_tablet_manager.cc:1918] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:44:38.069743 16841 log.cc:1196] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:44:38.070076 16841 ts_tablet_manager.cc:1939] T 2a2db53fea97428ab4440ba1f5d1c57d P 56ef361e784a4e09bc961b68451cae8a: Deleting consensus metadata
I20260924 04:44:38.071318 16334 catalog_manager.cc:4841] TS 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): tablet 2a2db53fea97428ab4440ba1f5d1c57d (table test-table [id=31969cfb6da7497ca8fec55b10496533]) successfully deleted
W20260924 04:44:38.075304 16337 catalog_manager.cc:4584] TS 35e8b5562b714b9b90ae39bfb9e40be0 (127.0.98.1:34297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a2db53fea97428ab4440ba1f5d1c57d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34297: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 04:44:38.077324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16679
I20260924 04:44:38.082355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16317
I20260924 04:44:58.089455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44145
--webserver_interface=127.0.98.62
--webserver_port=37263
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:44:58.209352 16843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:58.209657 16843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:58.209699 16843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:58.214010 16843 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:44:58.214087 16843 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:44:58.214104 16843 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:44:58.214118 16843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:44:58.214138 16843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:58.214155 16843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:58.214171 16843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:44:58.214238 16843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:44:58.219965 16843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:58.221168 16843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:58.222280 16843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:58.231956 16848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:58.232040 16843 server_base.cc:1034] running on GCE node
W20260924 04:44:58.231963 16849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:58.231976 16851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:58.232957 16843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:58.234228 16843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:58.235452 16843 hybrid_clock.cc:648] HybridClock initialized: now 1790225098235424 us; error 45 us; skew 500 ppm
Sep 24 04:44:58 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:58.243724 16843 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:44:58.244760 16843 webserver.cc:460] Webserver started at http://127.0.98.62:37263/ using document root <none> and password file <none>
I20260924 04:44:58.245316 16843 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:58.245378 16843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:58.249557 16843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:44:58.251878 16858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.253310 16843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:58.253480 16843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "34a89057b31f4a3fa635de52a272bd34"
format_stamp: "Formatted at 2026-09-24 04:44:15 on dist-test-slave-npg2"
server_key: "973530e5a9d5e9a28c0d2be6b8f38f0a"
server_key_iv: "cff2f7ca017289c386a6c94278ce3f89"
server_key_version: "encryptionkey@0"
I20260924 04:44:58.254106 16843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:58.268774 16843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:58.272233 16843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:58.272512 16843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:58.279860 16843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44145
I20260924 04:44:58.280156 16910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44145 every 8 connection(s)
I20260924 04:44:58.280993 16843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:44:58.285420 16911 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:44:58.286813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16843
I20260924 04:44:58.287333 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap starting.
I20260924 04:44:58.287330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34297
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44145
--webserver_port=43275
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:44:58.305871 16911 log.cc:824] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:58.317909 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:44:58.318341 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: Bootstrap complete.
I20260924 04:44:58.322722 16911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:58.323208 16911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a89057b31f4a3fa635de52a272bd34, State: Initialized, Role: FOLLOWER
I20260924 04:44:58.323778 16911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:58.323940 16911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:44:58.324033 16911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:44:58.324163 16911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:44:58.326195 16911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:58.326561 16911 leader_election.cc:302] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34a89057b31f4a3fa635de52a272bd34; no voters:
I20260924 04:44:58.326944 16911 leader_election.cc:288] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:44:58.327039 16915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:44:58.327322 16915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [term 3 LEADER]: Becoming Leader. State: Replica: 34a89057b31f4a3fa635de52a272bd34, State: Running, Role: LEADER
I20260924 04:44:58.327687 16915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } }
I20260924 04:44:58.328583 16911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:44:58.329411 16917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a89057b31f4a3fa635de52a272bd34. Latest consensus state: current_term: 3 leader_uuid: "34a89057b31f4a3fa635de52a272bd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } } }
I20260924 04:44:58.329531 16917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: This master's current role is: LEADER
I20260924 04:44:58.329468 16916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34a89057b31f4a3fa635de52a272bd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a89057b31f4a3fa635de52a272bd34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44145 } } }
I20260924 04:44:58.329588 16916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34 [sys.catalog]: This master's current role is: LEADER
I20260924 04:44:58.334137 16930 catalog_manager.cc:1259] Loaded cluster ID: fe74404dfcb541aca101d3a63910352d
I20260924 04:44:58.334210 16930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: loading cluster ID for follower catalog manager: success
I20260924 04:44:58.336954 16930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34a89057b31f4a3fa635de52a272bd34: acquiring CA information for follower catalog manager: success
F20260924 04:44:58.338258 16930 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f80b15f29ad google::LogMessage::Fail() at ??:0
@ 0x7f80b15f588a google::LogMessage::SendToLog() at ??:0
@ 0x7f80b15f24de google::LogMessage::Flush() at ??:0
@ 0x7f80b15f3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f80b42b618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f80bbf57989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f80bbf34023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f80bbf3454b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f80bbf2d652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f80bbf2c9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f80bbf73541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f80b3835512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f80b4e796db start_thread at ??:0
@ 0x7f80b00f061f clone at ??:0
W20260924 04:44:58.427821 16913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:58.428103 16913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:58.428148 16913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:58.432344 16913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:58.432418 16913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:58.432478 16913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:44:58.438082 16913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43275
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:58.439266 16913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:58.440366 16913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:58.450282 16951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:58.450313 16954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:58.450420 16952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:58.450677 16913 server_base.cc:1034] running on GCE node
I20260924 04:44:58.451197 16913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:58.451969 16913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:58.453233 16913 hybrid_clock.cc:648] HybridClock initialized: now 1790225098453200 us; error 51 us; skew 500 ppm
Sep 24 04:44:58 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:58.456200 16913 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:44:58.457186 16913 webserver.cc:460] Webserver started at http://127.0.98.1:43275/ using document root <none> and password file <none>
I20260924 04:44:58.457796 16913 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:58.457866 16913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:58.462109 16913 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:44:58.464694 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.466077 16913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:44:58.466233 16913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "35e8b5562b714b9b90ae39bfb9e40be0"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "9d0d6e5a4a9bc857baf32b349e7f073a"
server_key_iv: "9efcf7e6a8be67932462dcdd3011562d"
server_key_version: "encryptionkey@0"
I20260924 04:44:58.466811 16913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:58.495011 16913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:58.499010 16913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:58.499253 16913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:58.499898 16913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:58.501099 16968 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:44:58.507102 16913 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:44:58.507364 16913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260924 04:44:58.507480 16913 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:44:58.509400 16913 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:44:58.509531 16913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:44:58.509804 16968 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap starting.
I20260924 04:44:58.517467 16968 log.cc:824] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:44:58.521080 16913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34297
I20260924 04:44:58.521309 17076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34297 every 8 connection(s)
I20260924 04:44:58.522080 16913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:44:58.525024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 16913
I20260924 04:44:58.525462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37691
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44145
--webserver_port=33293
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:44:58.528473 16968 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:44:58.528961 16968 tablet_bootstrap.cc:492] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Bootstrap complete.
I20260924 04:44:58.529340 16968 ts_tablet_manager.cc:1397] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20260924 04:44:58.532512 16968 raft_consensus.cc:348] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:58.533088 16968 raft_consensus.cc:729] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8b5562b714b9b90ae39bfb9e40be0, State: Initialized, Role: FOLLOWER
I20260924 04:44:58.533661 16968 consensus_queue.cc:260] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:44:58.534830 16968 ts_tablet_manager.cc:1428] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 04:44:58.554700 17079 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:44:58.524171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:53633)
0924 04:44:58.524662 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.525918 (+ 1256us) client_negotiation.cc:174] Beginning negotiation
0924 04:44:58.527349 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:44:58.553843 (+ 26494us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":153,"threads_started":1}
W20260924 04:44:58.555335 17077 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44145 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44145: Client connection negotiation failed: client connection to 127.0.98.62:44145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:44:58.556092 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.555843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:45095)
0924 04:44:58.555976 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.556039 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:44:58.556735 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.556547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:33051)
0924 04:44:58.556650 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.556691 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:44:58.556964 17077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:44:58.647311 17080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:58.647590 17080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:58.647668 17080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:58.651868 17080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:58.651935 17080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:58.652027 17080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:44:58.657755 17080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33293
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:58.658946 17080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:58.660038 17080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:58.670145 17087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:58.670145 17086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:58.670178 17080 server_base.cc:1034] running on GCE node
W20260924 04:44:58.670476 17089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:58.670887 17080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:58.671657 17080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:58.672820 17080 hybrid_clock.cc:648] HybridClock initialized: now 1790225098672785 us; error 52 us; skew 500 ppm
Sep 24 04:44:58 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:58.675871 17080 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:44:58.676841 17080 webserver.cc:460] Webserver started at http://127.0.98.2:33293/ using document root <none> and password file <none>
I20260924 04:44:58.677417 17080 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:58.677479 17080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:58.681545 17080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:44:58.683776 17096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.685108 17080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:58.685202 17080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "72d8c13fd0924ebf965b02adca303ed6"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "a787ce507ab1f488b1ec99fd9e12dcd3"
server_key_iv: "bcdb20467e4dd848407604dfd9bdc2c2"
server_key_version: "encryptionkey@0"
I20260924 04:44:58.685874 17080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:58.717139 17080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:58.720816 17080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:58.721053 17080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:58.721690 17080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:58.722565 17080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:58.722612 17080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.722653 17080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:58.722711 17080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.732173 17080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37691
I20260924 04:44:58.732218 17209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37691 every 8 connection(s)
I20260924 04:44:58.733325 17080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:44:58.734107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17080
I20260924 04:44:58.734521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32891
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44145
--webserver_port=38399
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:44:58.737974 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.735920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:45341)
0924 04:44:58.736481 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.736638 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
W20260924 04:44:58.738564 17210 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44145 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44145: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
I20260924 04:44:58.739380 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.739049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:53055)
0924 04:44:58.739229 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.739296 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:44:58.740206 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.739926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:50499)
0924 04:44:58.740062 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.740127 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:44:58.740442 17210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:44:58.854296 17212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:44:58.854607 17212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:44:58.854657 17212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:44:58.859350 17212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:44:58.859422 17212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:44:58.859489 17212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:44:58.865212 17212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38399
--tserver_master_addrs=127.0.98.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:44:58.866492 17212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:44:58.867563 17212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:44:58.877915 17219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:58.877915 17218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:44:58.878003 17221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:44:58.878147 17212 server_base.cc:1034] running on GCE node
I20260924 04:44:58.878748 17212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:44:58.879458 17212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:44:58.880642 17212 hybrid_clock.cc:648] HybridClock initialized: now 1790225098880605 us; error 52 us; skew 500 ppm
Sep 24 04:44:58 dist-test-slave-npg2 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:44:58.883661 17212 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:44:58.884606 17212 webserver.cc:460] Webserver started at http://127.0.98.3:38399/ using document root <none> and password file <none>
I20260924 04:44:58.885231 17212 fs_manager.cc:359] Metadata directory not provided
I20260924 04:44:58.885334 17212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:44:58.889604 17212 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:44:58.892019 17228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.893564 17212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:44:58.893721 17212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "56ef361e784a4e09bc961b68451cae8a"
format_stamp: "Formatted at 2026-09-24 04:44:16 on dist-test-slave-npg2"
server_key: "033c594f590d8457633c66affca04b6e"
server_key_iv: "170628b82718789767fa8b8bbfeb94ca"
server_key_version: "encryptionkey@0"
I20260924 04:44:58.894326 17212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:44:58.920051 17212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:44:58.923614 17212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:44:58.923830 17212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:44:58.924467 17212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:44:58.925333 17212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:44:58.925405 17212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.925486 17212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:44:58.925540 17212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:44:58.934988 17212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32891
I20260924 04:44:58.935017 17341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32891 every 8 connection(s)
I20260924 04:44:58.936098 17212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:44:58.940408 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.938485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:49797)
0924 04:44:58.938968 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.939081 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
W20260924 04:44:58.940980 17342 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44145 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44145: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
I20260924 04:44:58.941663 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.941417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:32795)
0924 04:44:58.941561 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.941609 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 04:44:58.942345 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:58.942177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:59367)
0924 04:44:58.942262 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:58.942302 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:44:58.942589 17342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:44:58.943312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17212
I20260924 04:44:59.558084 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:59.557603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:35487)
0924 04:44:59.557918 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:59.557984 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
I20260924 04:44:59.741719 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:59.741194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:33907)
0924 04:44:59.741513 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:59.741569 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260924 04:44:59.943693 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:44:59.943272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:33299)
0924 04:44:59.943529 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:44:59.943587 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260924 04:45:00.262703 17349 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:00.262914 17349 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:45:00.264135 17349 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
Sep 24 04:45:00 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 04:45:00 dist-test-slave-npg2 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 04:45:00.279655 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.264559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60053 (local address 127.0.98.2:37691)
0924 04:45:00.265053 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:00.265070 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:00.265118 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:45:00.265839 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:00.265847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:00.266244 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:00.266497 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:00.267957 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.268955 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.270264 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.270734 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.270855 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:00.274141 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:00.274149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:00.274169 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:00.274210 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:00.276732 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:00.277465 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:00.277469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:00.277474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:00.277598 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:00.277953 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:00.277956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:00.277959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:00.278403 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:00.278571 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:00.279103 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 04:45:00.279440 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":83,"threads_started":1}
I20260924 04:45:00.279659 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.264745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32891 (local address 127.0.98.1:42237)
0924 04:45:00.265026 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:00.265036 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:00.265161 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:00.266715 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:00.266719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:00.267238 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:00.267777 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:00.267792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.269200 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.269203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:00.270104 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:00.270113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.270953 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.270956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:00.271062 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:00.271863 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:00.271889 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:00.274320 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:00.276838 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:00.276848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:00.276866 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:00.277275 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:00.277683 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:00.277690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:00.277695 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:00.277860 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:00.278521 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:00.278528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:00.278789 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:45:00.278976 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 04:45:00.279345 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":105,"spinlock_wait_cycles":30336,"thread_start_us":69,"threads_started":1}
I20260924 04:45:00.279798 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.264934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42237 (local address 127.0.98.3:32891)
0924 04:45:00.265267 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:00.265291 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:00.265332 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:45:00.266034 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:00.266040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:00.266450 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:00.266746 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:00.267947 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.269053 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.270230 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.270828 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.270987 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:00.274538 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:00.274543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:00.274555 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:00.274583 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:00.276698 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:00.277433 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:00.277439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:00.277446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:00.277577 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:00.277988 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:00.277990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:00.277992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:00.278396 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:00.278635 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:00.279327 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 04:45:00.279606 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260924 04:45:00.279670 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.264405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37691 (local address 127.0.98.1:60053)
0924 04:45:00.264641 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:00.264661 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:00.264766 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:00.266587 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:00.266597 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:00.267239 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:00.267776 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:00.267793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.269096 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.269102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:00.270109 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:00.270118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:00.270854 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:00.270861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:00.271057 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:00.271861 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:00.271877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:00.273874 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:00.276888 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:00.276890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:00.276892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:00.277265 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:00.277723 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:00.277726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:00.277727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:00.277852 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:00.278635 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:00.278638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:00.278788 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 04:45:00.278975 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 04:45:00.279440 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":311,"spinlock_wait_cycles":12288,"thread_start_us":79,"threads_started":1}
W20260924 04:45:00.283543 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
W20260924 04:45:00.283840 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:45:00.283957 16962 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0; no voters: 56ef361e784a4e09bc961b68451cae8a, 72d8c13fd0924ebf965b02adca303ed6
I20260924 04:45:00.284229 17349 raft_consensus.cc:2726] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:45:00.559533 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.559061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:59807)
0924 04:45:00.559292 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:00.559444 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260924 04:45:00.743064 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.742618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:32895)
0924 04:45:00.742891 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:00.742952 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260924 04:45:00.945017 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:00.944576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:37373)
0924 04:45:00.944857 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:00.944918 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260924 04:45:01.560773 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:01.560347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:37813)
0924 04:45:01.560620 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:01.560676 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260924 04:45:01.744532 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:01.743971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:51065)
0924 04:45:01.744270 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:01.744427 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":76,"threads_started":1}
I20260924 04:45:01.815507 17358 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:01.815636 17358 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:45:01.816027 17358 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
W20260924 04:45:01.817013 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
W20260924 04:45:01.817169 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:45:01.817231 16962 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0; no voters: 56ef361e784a4e09bc961b68451cae8a, 72d8c13fd0924ebf965b02adca303ed6
I20260924 04:45:01.817456 17358 raft_consensus.cc:2726] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:45:01.946496 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:01.945956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:58891)
0924 04:45:01.946306 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:01.946378 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260924 04:45:02.562163 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:02.561639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:41965)
0924 04:45:02.561979 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:02.562051 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":88,"threads_started":1}
I20260924 04:45:02.746146 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:02.745507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:44683)
0924 04:45:02.745927 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:02.746015 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":127,"threads_started":1}
I20260924 04:45:02.947718 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:02.947270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:36865)
0924 04:45:02.947559 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:02.947620 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260924 04:45:03.504161 17363 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:03.504346 17363 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:45:03.504841 17363 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
W20260924 04:45:03.505748 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
W20260924 04:45:03.505932 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:45:03.505993 16962 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0; no voters: 56ef361e784a4e09bc961b68451cae8a, 72d8c13fd0924ebf965b02adca303ed6
I20260924 04:45:03.506170 17363 raft_consensus.cc:2726] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:45:03.564136 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:03.563581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:35361)
0924 04:45:03.563963 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:03.564033 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":82,"threads_started":1}
I20260924 04:45:03.747721 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:03.747110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:37829)
0924 04:45:03.747552 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:03.747619 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":94,"threads_started":1}
I20260924 04:45:03.949198 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:03.948697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:56877)
0924 04:45:03.949025 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:03.949094 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
I20260924 04:45:04.565807 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:04.565329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:55409)
0924 04:45:04.565643 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:04.565710 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260924 04:45:04.749120 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:04.748665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:45851)
0924 04:45:04.748951 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:04.749016 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260924 04:45:04.950683 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:04.950188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:57099)
0924 04:45:04.950513 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:04.950582 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20260924 04:45:05.567184 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:05.566688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:53097)
0924 04:45:05.566958 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:05.567020 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260924 04:45:05.750700 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:05.750178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:43777)
0924 04:45:05.750537 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:05.750602 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":79,"threads_started":1}
I20260924 04:45:05.952100 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:05.951582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:33925)
0924 04:45:05.951942 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:05.952003 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260924 04:45:06.568537 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:06.568027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:48211)
0924 04:45:06.568359 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:06.568424 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260924 04:45:06.752030 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:06.751562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:41165)
0924 04:45:06.751861 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:06.751933 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260924 04:45:06.953701 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:06.952970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:57455)
0924 04:45:06.953518 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:06.953593 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260924 04:45:07.569818 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:07.569362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:33813)
0924 04:45:07.569656 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:07.569722 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260924 04:45:07.753310 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:07.752877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:38125)
0924 04:45:07.753145 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:07.753208 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260924 04:45:07.955017 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:07.954533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:43063)
0924 04:45:07.954856 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:07.954907 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260924 04:45:08.289996 17379 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:08.290166 17379 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:45:08.290709 17379 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
W20260924 04:45:08.291572 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
W20260924 04:45:08.291721 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:45:08.291787 16962 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0; no voters: 56ef361e784a4e09bc961b68451cae8a, 72d8c13fd0924ebf965b02adca303ed6
I20260924 04:45:08.291960 17379 raft_consensus.cc:2726] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:45:08.571534 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:08.570932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:56917)
0924 04:45:08.571320 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:08.571433 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260924 04:45:08.754700 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:08.754207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:32981)
0924 04:45:08.754542 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:08.754604 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":74,"threads_started":1}
I20260924 04:45:08.956555 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:08.956082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:57145)
0924 04:45:08.956379 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:08.956452 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":76,"threads_started":1}
I20260924 04:45:09.572719 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:09.572303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:45453)
0924 04:45:09.572568 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:09.572626 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260924 04:45:09.756109 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:09.755645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:59759)
0924 04:45:09.755924 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:09.756007 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260924 04:45:09.958009 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:09.957462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:59461)
0924 04:45:09.957835 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:09.957913 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
I20260924 04:45:10.069446 17386 raft_consensus.cc:482] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:10.069628 17386 raft_consensus.cc:504] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d8c13fd0924ebf965b02adca303ed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37691 } } peers { permanent_uuid: "56ef361e784a4e09bc961b68451cae8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "35e8b5562b714b9b90ae39bfb9e40be0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34297 } }
I20260924 04:45:10.070117 17386 leader_election.cc:288] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691), 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891)
W20260924 04:45:10.071090 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72d8c13fd0924ebf965b02adca303ed6 (127.0.98.2:37691): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
W20260924 04:45:10.071302 16962 leader_election.cc:341] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ef361e784a4e09bc961b68451cae8a (127.0.98.3:32891): Not found: Tablet not found: 2a2db53fea97428ab4440ba1f5d1c57d
I20260924 04:45:10.071380 16962 leader_election.cc:302] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35e8b5562b714b9b90ae39bfb9e40be0; no voters: 56ef361e784a4e09bc961b68451cae8a, 72d8c13fd0924ebf965b02adca303ed6
I20260924 04:45:10.071576 17386 raft_consensus.cc:2726] T 2a2db53fea97428ab4440ba1f5d1c57d P 35e8b5562b714b9b90ae39bfb9e40be0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:45:10.574111 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:10.573625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:39829)
0924 04:45:10.573921 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:10.574010 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260924 04:45:10.757282 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:10.756869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:38679)
0924 04:45:10.757127 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:10.757185 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260924 04:45:10.959467 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:10.958961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:43027)
0924 04:45:10.959289 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:10.959351 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":86,"threads_started":1}
I20260924 04:45:11.575457 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:11.574852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:36849)
0924 04:45:11.575236 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:11.575294 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":125,"threads_started":1}
I20260924 04:45:11.758632 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:11.758156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:46199)
0924 04:45:11.758466 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:11.758534 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260924 04:45:11.960691 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:11.960265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:39995)
0924 04:45:11.960535 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:11.960593 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260924 04:45:12.576771 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:12.576329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:59967)
0924 04:45:12.576598 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:12.576664 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260924 04:45:12.760092 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:12.759634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:45779)
0924 04:45:12.759914 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:12.759971 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260924 04:45:12.962082 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:12.961667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:36321)
0924 04:45:12.961924 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:12.961981 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260924 04:45:13.578297 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:13.577881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.1:54987)
0924 04:45:13.578151 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:13.578203 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260924 04:45:13.761587 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:13.761079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.2:47123)
0924 04:45:13.761383 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:13.761444 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
I20260924 04:45:13.953015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 16913
I20260924 04:45:13.959195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17080
I20260924 04:45:13.963685 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:13.963079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44145 (local address 127.0.98.3:40427)
0924 04:45:13.963438 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:13.963523 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":123,"threads_started":1}
I20260924 04:45:13.965119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17212
2026-09-24T04:45:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59493 ms)
[----------] 30 tests from SecurityITest (248661 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:14 dist-test-slave-npg2 krb5kdc[17401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:14 dist-test-slave-npg2 krb5kdc[17401](info): set up 2 sockets
Sep 24 04:45:14 dist-test-slave-npg2 krb5kdc[17401](info): commencing operation
krb5kdc: starting...
W20260924 04:45:15.051010 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:15.174784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:15.289006 17414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:15.289341 17414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:15.289440 17414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:15.293684 17414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:15.293783 17414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:15.293817 17414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:15.293857 17414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:15.293888 17414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:15.299374 17414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:15.300518 17414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:15.301553 17414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:15.311095 17422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:15.311151 17420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:15.311342 17414 server_base.cc:1034] running on GCE node
W20260924 04:45:15.311054 17419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:15.312074 17414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:15.313225 17414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:15.314508 17414 hybrid_clock.cc:648] HybridClock initialized: now 1790225115314493 us; error 29 us; skew 500 ppm
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:15.317498 17414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:15.318424 17414 webserver.cc:460] Webserver started at http://127.0.98.62:43045/ using document root <none> and password file <none>
I20260924 04:45:15.318957 17414 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:15.319017 17414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:15.319258 17414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:15.321358 17414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "b1890ddd15a44dd2ab94efb904650fcd"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "5508d79830343fbe8d10e1e055616043"
server_key_iv: "86aa32b42239c7c6c72dfae699da15d0"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.321900 17414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1890ddd15a44dd2ab94efb904650fcd"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "5508d79830343fbe8d10e1e055616043"
server_key_iv: "86aa32b42239c7c6c72dfae699da15d0"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.326772 17414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 04:45:15.329533 17429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.330896 17414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:15.331084 17414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "b1890ddd15a44dd2ab94efb904650fcd"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "5508d79830343fbe8d10e1e055616043"
server_key_iv: "86aa32b42239c7c6c72dfae699da15d0"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.331292 17414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:15.359232 17414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:15.363272 17414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:15.363482 17414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:15.370918 17414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32889
I20260924 04:45:15.370934 17481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32889 every 8 connection(s)
I20260924 04:45:15.371990 17414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:15.374815 17482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:15.380462 17482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: Bootstrap starting.
I20260924 04:45:15.381922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17414
I20260924 04:45:15.382058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:15.382364 392 external_mini_cluster.cc:1428] Setting key 7f22fdb21a1e1594a73acbca7f4b4a69
I20260924 04:45:15.383431 17482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:15.384310 17482 log.cc:824] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:15.386395 17482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: No bootstrap required, opened a new log
I20260924 04:45:15.390431 17482 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1890ddd15a44dd2ab94efb904650fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260924 04:45:15.390683 17482 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:15.390777 17482 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1890ddd15a44dd2ab94efb904650fcd, State: Initialized, Role: FOLLOWER
I20260924 04:45:15.391360 17482 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1890ddd15a44dd2ab94efb904650fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:15.391526 17482 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:15.391592 17482 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:15.391692 17482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:15.392838 17482 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1890ddd15a44dd2ab94efb904650fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260924 04:45:15.393226 17482 leader_election.cc:302] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1890ddd15a44dd2ab94efb904650fcd; no voters:
I20260924 04:45:15.393589 17482 leader_election.cc:288] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:15.393705 17487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:15.393909 17487 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [term 1 LEADER]: Becoming Leader. State: Replica: b1890ddd15a44dd2ab94efb904650fcd, State: Running, Role: LEADER
I20260924 04:45:15.394336 17487 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1890ddd15a44dd2ab94efb904650fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260924 04:45:15.395004 17482 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:15.396165 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1890ddd15a44dd2ab94efb904650fcd. Latest consensus state: current_term: 1 leader_uuid: "b1890ddd15a44dd2ab94efb904650fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1890ddd15a44dd2ab94efb904650fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260924 04:45:15.396294 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:15.396806 17488 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1890ddd15a44dd2ab94efb904650fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1890ddd15a44dd2ab94efb904650fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260924 04:45:15.396904 17488 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:15.397796 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:15.383640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.0.98.62:32889)
0924 04:45:15.384108 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:15.384124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:15.384158 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:45:15.385030 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:15.385057 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:15.385404 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:15.385758 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:15.386557 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.387640 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.388536 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.389086 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.389267 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:15.392165 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:15.392170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:15.392184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:15.392209 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:15.394225 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:15.394733 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:15.394739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:15.394744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:15.394881 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:15.395232 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:15.395236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:15.395238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:15.395786 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:15.396005 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:15.396461 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 04:45:15.396795 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":142,"threads_started":1}
W20260924 04:45:15.400282 17502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:15.400360 17502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:15.400432 17496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:15.401268 17496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:15.406823 17496 catalog_manager.cc:1347] Generated new cluster ID: 4b285d7534404445943656e22267a9b2
I20260924 04:45:15.406906 17496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:15.415475 17496 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:15.416781 17496 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:15.428789 17496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: Generated new TSK 0
I20260924 04:45:15.429725 17496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:45:15.467252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:15.585669 17508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:15.585990 17508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:15.586038 17508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:15.590497 17508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:15.590584 17508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:15.590648 17508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:45:15.596793 17508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:15.597904 17508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:15.598901 17508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:15.609655 17514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:15.609557 17513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:15.609594 17516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:15.610090 17508 server_base.cc:1034] running on GCE node
I20260924 04:45:15.610531 17508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:15.611361 17508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:15.612534 17508 hybrid_clock.cc:648] HybridClock initialized: now 1790225115612491 us; error 56 us; skew 500 ppm
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:15.615688 17508 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:45:15.616549 17508 webserver.cc:460] Webserver started at http://127.0.98.1:45837/ using document root <none> and password file <none>
I20260924 04:45:15.617086 17508 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:15.617142 17508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:15.617311 17508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:15.619386 17508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "e83a072f08014cb5bc6ee55ac0f4ab84"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "7a35121f2ecb7759d048c07b11320a41"
server_key_iv: "072024abc08d28e6d91ed441ed2ede6e"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.619890 17508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e83a072f08014cb5bc6ee55ac0f4ab84"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "7a35121f2ecb7759d048c07b11320a41"
server_key_iv: "072024abc08d28e6d91ed441ed2ede6e"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.623752 17508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:45:15.626272 17523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.627717 17508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:15.627815 17508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "e83a072f08014cb5bc6ee55ac0f4ab84"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "7a35121f2ecb7759d048c07b11320a41"
server_key_iv: "072024abc08d28e6d91ed441ed2ede6e"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.627907 17508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:15.647555 17508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:15.651288 17508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:15.651503 17508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:15.652143 17508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:15.652987 17508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:15.653033 17508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.653110 17508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:15.653141 17508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.662233 17508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44691
I20260924 04:45:15.662276 17636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44691 every 8 connection(s)
I20260924 04:45:15.663177 17508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:45:15.663932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17508
I20260924 04:45:15.664077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:45:15.664367 392 external_mini_cluster.cc:1428] Setting key 501f383504e15d73fa62ea513b18206b
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:45:15.677681 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:15.665224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40509 (local address 127.0.98.62:32889)
0924 04:45:15.665366 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:15.665372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:15.666441 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 04:45:15.667537 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:15.667540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:15.667595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:15.667673 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:15.668851 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.669390 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.670593 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.670836 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.670953 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:15.673821 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:15.673824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:15.673826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:15.673851 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:15.675506 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:15.676066 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:15.676070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:15.676072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:15.676152 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:15.676509 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:15.676512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:15.676514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:15.676702 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:15.676808 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:15.677382 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 04:45:15.677514 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:45:15.678898 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:15.665736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.1:40509)
0924 04:45:15.666243 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:15.666310 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:15.667328 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:15.667846 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:15.667853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:15.668213 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:15.668700 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:15.668714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.669523 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.669526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:15.670448 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:15.670457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.670995 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.670999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:15.671231 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:15.671860 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:15.671877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:15.673634 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:15.675679 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:15.675684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:15.675697 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:15.675947 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:15.676254 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:15.676258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:15.676260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:15.676404 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:15.676808 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:15.676816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:15.677126 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 04:45:15.677341 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 04:45:15.677707 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":138,"threads_started":1}
I20260924 04:45:15.680311 17637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260924 04:45:15.680676 17637 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:15.681185 17637 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260924 04:45:15.682729 17446 ts_manager.cc:194] Registered new tserver with Master: e83a072f08014cb5bc6ee55ac0f4ab84 (127.0.98.1:44691)
I20260924 04:45:15.683947 17446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:45:15.696002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:15.812907 17642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:15.813216 17642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:15.813300 17642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:15.817647 17642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:15.817749 17642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:15.817840 17642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:45:15.823577 17642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:15.824762 17642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:15.825835 17642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:15.835866 17648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:15.835865 17647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:15.835879 17650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:15.836346 17642 server_base.cc:1034] running on GCE node
I20260924 04:45:15.836700 17642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:15.837342 17642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:15.838506 17642 hybrid_clock.cc:648] HybridClock initialized: now 1790225115838489 us; error 38 us; skew 500 ppm
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:15.841375 17642 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:45:15.842299 17642 webserver.cc:460] Webserver started at http://127.0.98.2:37197/ using document root <none> and password file <none>
I20260924 04:45:15.842862 17642 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:15.842921 17642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:15.843214 17642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:15.845238 17642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "888095a5485741368825e53d459766ac"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "01cad63580e80e0bb84140f3328bf753"
server_key_iv: "57d2122a4f5891ffd93b8dac2aa50cfa"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.845791 17642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "888095a5485741368825e53d459766ac"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "01cad63580e80e0bb84140f3328bf753"
server_key_iv: "57d2122a4f5891ffd93b8dac2aa50cfa"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.849668 17642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 04:45:15.852305 17657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.853545 17642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:15.853705 17642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "888095a5485741368825e53d459766ac"
format_stamp: "Formatted at 2026-09-24 04:45:15 on dist-test-slave-npg2"
server_key: "01cad63580e80e0bb84140f3328bf753"
server_key_iv: "57d2122a4f5891ffd93b8dac2aa50cfa"
server_key_version: "encryptionkey@0"
I20260924 04:45:15.853824 17642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:15.871435 17642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:15.874558 17642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:15.874779 17642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:15.875460 17642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:15.876328 17642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:15.876399 17642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.876477 17642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:15.876519 17642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:15.885774 17642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41375
I20260924 04:45:15.885833 17770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41375 every 8 connection(s)
I20260924 04:45:15.886795 17642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:45:15.893167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17642
I20260924 04:45:15.893299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:45:15.893630 392 external_mini_cluster.cc:1428] Setting key 2be0fc1faac22421926b6ad918a1dd79
Sep 24 04:45:15 dist-test-slave-npg2 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:15.902842 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:15.888956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39615 (local address 127.0.98.62:32889)
0924 04:45:15.889114 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:15.889117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:15.890083 (+ 966us) server_negotiation.cc:407] Connection header received
0924 04:45:15.890955 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:15.890958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:15.891006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:15.891082 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:15.892396 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.892981 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.894682 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.895017 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.895193 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:15.898497 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:15.898501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:15.898503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:15.898528 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:15.900092 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:15.900883 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:15.900886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:15.900887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:15.901031 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:15.901407 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:15.901413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:15.901414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:15.901696 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:15.901820 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:15.902427 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 04:45:15.902597 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:45:15.903604 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:15.889290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.2:39615)
0924 04:45:15.889862 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:15.889945 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:15.890742 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:15.891281 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:15.891289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:15.891732 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:15.892241 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:15.892255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.893122 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.893126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:15.894468 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:15.894553 (+ 85us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:15.895207 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:15.895211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:15.895421 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:15.896231 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:15.896258 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:15.898287 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:15.900329 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:15.900339 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:15.900371 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:15.900758 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:15.901158 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:15.901160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:15.901161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:15.901304 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:15.901823 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:15.901829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:15.902134 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 04:45:15.902310 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 04:45:15.902611 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
I20260924 04:45:15.904826 17771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260924 04:45:15.905117 17771 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:45:15.905745 17771 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260924 04:45:15.906827 17446 ts_manager.cc:194] Registered new tserver with Master: 888095a5485741368825e53d459766ac (127.0.98.2:41375)
I20260924 04:45:15.907579 17446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:45:15.926839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:16.043036 17776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:16.043402 17776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:16.043497 17776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:16.047804 17776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:16.047870 17776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:16.047967 17776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:45:16.053391 17776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:16.054653 17776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:16.055781 17776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:16.065881 17781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:16.065881 17782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:16.065881 17784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:16.066236 17776 server_base.cc:1034] running on GCE node
I20260924 04:45:16.066680 17776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:16.067389 17776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:16.068640 17776 hybrid_clock.cc:648] HybridClock initialized: now 1790225116068602 us; error 52 us; skew 500 ppm
Sep 24 04:45:16 dist-test-slave-npg2 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:16.071609 17776 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:45:16.072598 17776 webserver.cc:460] Webserver started at http://127.0.98.3:36221/ using document root <none> and password file <none>
I20260924 04:45:16.073215 17776 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:16.073307 17776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:16.073572 17776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:16.075770 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "486c78f3f9334a629ff31637a50a87eb"
format_stamp: "Formatted at 2026-09-24 04:45:16 on dist-test-slave-npg2"
server_key: "03f71c3413f5e5797b9ae9b7170c5e5a"
server_key_iv: "b0dbb840bf9645fcbe8768ae919b65a9"
server_key_version: "encryptionkey@0"
I20260924 04:45:16.076342 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "486c78f3f9334a629ff31637a50a87eb"
format_stamp: "Formatted at 2026-09-24 04:45:16 on dist-test-slave-npg2"
server_key: "03f71c3413f5e5797b9ae9b7170c5e5a"
server_key_iv: "b0dbb840bf9645fcbe8768ae919b65a9"
server_key_version: "encryptionkey@0"
I20260924 04:45:16.080633 17776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:45:16.083353 17791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:16.084687 17776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:45:16.084843 17776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "486c78f3f9334a629ff31637a50a87eb"
format_stamp: "Formatted at 2026-09-24 04:45:16 on dist-test-slave-npg2"
server_key: "03f71c3413f5e5797b9ae9b7170c5e5a"
server_key_iv: "b0dbb840bf9645fcbe8768ae919b65a9"
server_key_version: "encryptionkey@0"
I20260924 04:45:16.084970 17776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:16.109220 17776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:16.112461 17776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:16.112689 17776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:16.113308 17776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:16.114162 17776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:16.114234 17776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:16.114307 17776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:16.114351 17776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:16.123720 17776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43209
I20260924 04:45:16.123739 17904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43209 every 8 connection(s)
I20260924 04:45:16.124660 17776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:45:16.134006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17776
I20260924 04:45:16.134186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:45:16.134475 392 external_mini_cluster.cc:1428] Setting key 29dd361e39dfcf5351b0c39d3d267470
Sep 24 04:45:16 dist-test-slave-npg2 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:16.139505 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.126502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57683 (local address 127.0.98.62:32889)
0924 04:45:16.126656 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.126660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.127640 (+ 980us) server_negotiation.cc:407] Connection header received
0924 04:45:16.128958 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.128961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.129024 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.129150 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:16.130370 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.130958 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.132267 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.132492 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.132605 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.135694 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:16.135697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:16.135699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:16.135745 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:16.137423 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:16.137934 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:16.137936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:16.137938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:16.138015 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:16.138421 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:16.138423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:16.138424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:16.138603 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:16.138677 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.139278 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.139384 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:45:16.140533 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.126963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.3:57683)
0924 04:45:16.127433 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.127476 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:16.128723 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:16.129333 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:16.129340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:16.129723 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:16.130206 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.130219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.131083 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.131086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.132134 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.132144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.132648 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.132652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.132820 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.133456 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:16.133473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:16.135466 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:16.137529 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:16.137534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:16.137588 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:16.137849 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:16.138162 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:16.138165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:16.138167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:16.138324 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:16.138720 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:16.138725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:16.138986 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:45:16.139245 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 04:45:16.139570 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260924 04:45:16.141943 17905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260924 04:45:16.142174 17905 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:16.142644 17905 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260924 04:45:16.143714 17446 ts_manager.cc:194] Registered new tserver with Master: 486c78f3f9334a629ff31637a50a87eb (127.0.98.3:43209)
I20260924 04:45:16.144305 17446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57683
I20260924 04:45:16.148660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:45:16.158420 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.150745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.0.98.62:32889)
0924 04:45:16.150846 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.150873 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.151099 (+ 226us) server_negotiation.cc:407] Connection header received
0924 04:45:16.151263 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.151266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.151317 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.151403 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:16.152115 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.152629 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.153499 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.153725 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.153837 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.155280 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:16.155283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:16.155286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:16.155310 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:16.156787 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:16.157256 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:16.157259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:16.157260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:16.157333 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:16.157589 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:16.157591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:16.157592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:16.157811 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:16.157911 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.158199 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.158302 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:45:16.161273 17446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:45:16.164363 17446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:45:16.179196 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.173823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44691 (local address 127.0.0.1:42190)
0924 04:45:16.174418 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.174433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:16.174519 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:16.175323 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:16.175326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:16.175345 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:16.175431 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.175436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.176480 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.176483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.177745 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.177755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.178565 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.178569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.178710 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.178758 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 04:45:16.178877 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:45:16.179012 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":40,"thread_start_us":92,"threads_started":1}
I20260924 04:45:16.179261 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.174276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.98.1:44691)
0924 04:45:16.174826 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.174840 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.174862 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:45:16.174940 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.174944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.175171 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.175332 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:16.175551 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.176335 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.177853 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.178433 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.178565 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.178717 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.178931 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.179041 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":96,"threads_started":1}
I20260924 04:45:16.179520 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.173356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.0.98.3:43209)
0924 04:45:16.173695 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.173709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.173766 (+ 57us) server_negotiation.cc:407] Connection header received
0924 04:45:16.174084 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.174090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.174318 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.174510 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:16.175044 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.176280 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.177575 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.178431 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.178567 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.178716 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.179265 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.179369 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":149,"threads_started":1}
I20260924 04:45:16.179809 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.173160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43209 (local address 127.0.0.1:59112)
0924 04:45:16.173627 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.173690 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:16.173840 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:16.174724 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:16.174730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:16.174752 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:16.174906 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.174917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.176429 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.176434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.177442 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.177451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.178891 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.178894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.178990 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.179012 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:45:16.179081 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:45:16.179169 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":60,"thread_start_us":98,"threads_started":1}
I20260924 04:45:16.180380 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.173939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41375 (local address 127.0.0.1:43812)
0924 04:45:16.174748 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.174761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:16.174849 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:16.175496 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:16.175499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:16.175520 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:16.175625 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.175631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.177026 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.177029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.178726 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.178739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.179996 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.179999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.180082 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.180104 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:45:16.180178 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:45:16.180254 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":605,"thread_start_us":68,"threads_started":1}
I20260924 04:45:16.180747 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.174165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.0.98.2:41375)
0924 04:45:16.174579 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.174590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.174944 (+ 354us) server_negotiation.cc:407] Connection header received
0924 04:45:16.175012 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.175017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.175199 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.175332 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:16.175755 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.176849 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.178845 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.179883 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.180001 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.180143 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.180199 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.180593 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":104,"threads_started":1}
I20260924 04:45:16.182924 17839 tablet_service.cc:1467] Processing CreateTablet for tablet 514902199ffc414582f93c84c2c1500a (DEFAULT_TABLE table=test-table [id=f916db570b2749e6a5349624db68e56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:16.182924 17571 tablet_service.cc:1467] Processing CreateTablet for tablet 514902199ffc414582f93c84c2c1500a (DEFAULT_TABLE table=test-table [id=f916db570b2749e6a5349624db68e56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:16.184191 17839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514902199ffc414582f93c84c2c1500a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:16.184191 17571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514902199ffc414582f93c84c2c1500a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:16.186136 17705 tablet_service.cc:1467] Processing CreateTablet for tablet 514902199ffc414582f93c84c2c1500a (DEFAULT_TABLE table=test-table [id=f916db570b2749e6a5349624db68e56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:16.187003 17705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514902199ffc414582f93c84c2c1500a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:16.191880 17923 tablet_bootstrap.cc:492] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Bootstrap starting.
I20260924 04:45:16.192333 17922 tablet_bootstrap.cc:492] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: Bootstrap starting.
I20260924 04:45:16.194017 17924 tablet_bootstrap.cc:492] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Bootstrap starting.
I20260924 04:45:16.195010 17923 tablet_bootstrap.cc:654] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:16.195574 17922 tablet_bootstrap.cc:654] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:16.196352 17924 tablet_bootstrap.cc:654] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:16.196440 17923 log.cc:824] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:16.196532 17922 log.cc:824] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:16.197549 17924 log.cc:824] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:16.198580 17922 tablet_bootstrap.cc:492] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: No bootstrap required, opened a new log
I20260924 04:45:16.198625 17923 tablet_bootstrap.cc:492] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: No bootstrap required, opened a new log
I20260924 04:45:16.198792 17922 ts_tablet_manager.cc:1397] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:45:16.198792 17923 ts_tablet_manager.cc:1397] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 04:45:16.199455 17924 tablet_bootstrap.cc:492] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: No bootstrap required, opened a new log
I20260924 04:45:16.199630 17924 ts_tablet_manager.cc:1397] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:45:16.202884 17923 raft_consensus.cc:348] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.202935 17922 raft_consensus.cc:348] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.203212 17922 raft_consensus.cc:374] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:16.203213 17923 raft_consensus.cc:374] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:16.203270 17923 raft_consensus.cc:729] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486c78f3f9334a629ff31637a50a87eb, State: Initialized, Role: FOLLOWER
I20260924 04:45:16.203287 17922 raft_consensus.cc:729] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83a072f08014cb5bc6ee55ac0f4ab84, State: Initialized, Role: FOLLOWER
I20260924 04:45:16.203363 17924 raft_consensus.cc:348] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.203612 17924 raft_consensus.cc:374] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:16.203711 17924 raft_consensus.cc:729] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888095a5485741368825e53d459766ac, State: Initialized, Role: FOLLOWER
I20260924 04:45:16.203796 17923 consensus_queue.cc:260] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.203809 17922 consensus_queue.cc:260] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.204094 17924 consensus_queue.cc:260] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.204548 17637 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260924 04:45:16.204909 17922 ts_tablet_manager.cc:1428] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:45:16.205081 17924 ts_tablet_manager.cc:1428] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:45:16.205618 17771 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260924 04:45:16.206951 17923 ts_tablet_manager.cc:1428] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260924 04:45:16.207352 17905 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
W20260924 04:45:16.375612 17906 tablet.cc:2369] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:45:16.388118 17772 tablet.cc:2369] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:45:16.414801 17638 tablet.cc:2369] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:16.485877 17928 raft_consensus.cc:482] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:16.486126 17928 raft_consensus.cc:504] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.487373 17928 leader_election.cc:288] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486c78f3f9334a629ff31637a50a87eb (127.0.98.3:43209), 888095a5485741368825e53d459766ac (127.0.98.2:41375)
I20260924 04:45:16.492208 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.487868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37339 (local address 127.0.98.3:43209)
0924 04:45:16.488068 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.488073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.488243 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:45:16.488442 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.488446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.488506 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.488598 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:16.489016 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.489663 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.490899 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.491581 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.491681 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.491734 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.491978 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.492056 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 04:45:16.492205 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.487721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43209 (local address 127.0.98.1:37339)
0924 04:45:16.488106 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.488120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:16.488306 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:16.488656 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:16.488660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:16.488686 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:16.488869 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.488876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.489768 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.489771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.490761 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.490770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.491716 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.491720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.491834 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.491862 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:45:16.491949 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:45:16.492024 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":21,"thread_start_us":131,"threads_started":1}
I20260924 04:45:16.492367 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.488374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44385 (local address 127.0.98.2:41375)
0924 04:45:16.488526 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.488539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.488665 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:45:16.488836 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.488840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.488890 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.488970 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:16.489289 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.489956 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.491227 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.491747 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.491852 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.491918 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.492157 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.492244 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:45:16.492395 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.488257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41375 (local address 127.0.98.1:44385)
0924 04:45:16.488579 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.488594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:16.488675 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:16.489015 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:16.489018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:16.489037 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:16.489162 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.489169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.490089 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.490091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.491060 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:16.491070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.491864 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.491866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:16.492024 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.492047 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:45:16.492125 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:45:16.492193 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":25,"thread_start_us":96,"threads_started":1}
I20260924 04:45:16.493033 17725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514902199ffc414582f93c84c2c1500a" candidate_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888095a5485741368825e53d459766ac" is_pre_election: true
I20260924 04:45:16.493252 17859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514902199ffc414582f93c84c2c1500a" candidate_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486c78f3f9334a629ff31637a50a87eb" is_pre_election: true
I20260924 04:45:16.493382 17725 raft_consensus.cc:2454] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83a072f08014cb5bc6ee55ac0f4ab84 in term 0.
I20260924 04:45:16.493542 17859 raft_consensus.cc:2454] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83a072f08014cb5bc6ee55ac0f4ab84 in term 0.
I20260924 04:45:16.493965 17525 leader_election.cc:302] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888095a5485741368825e53d459766ac, e83a072f08014cb5bc6ee55ac0f4ab84; no voters:
I20260924 04:45:16.494267 17928 raft_consensus.cc:2781] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:45:16.494393 17928 raft_consensus.cc:482] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:45:16.494460 17928 raft_consensus.cc:3037] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:16.495676 17928 raft_consensus.cc:504] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.496039 17928 leader_election.cc:288] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [CANDIDATE]: Term 1 election: Requested vote from peers 486c78f3f9334a629ff31637a50a87eb (127.0.98.3:43209), 888095a5485741368825e53d459766ac (127.0.98.2:41375)
I20260924 04:45:16.496533 17725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514902199ffc414582f93c84c2c1500a" candidate_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888095a5485741368825e53d459766ac"
I20260924 04:45:16.496629 17859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514902199ffc414582f93c84c2c1500a" candidate_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486c78f3f9334a629ff31637a50a87eb"
I20260924 04:45:16.496678 17725 raft_consensus.cc:3037] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:16.496750 17859 raft_consensus.cc:3037] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:16.498029 17725 raft_consensus.cc:2454] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83a072f08014cb5bc6ee55ac0f4ab84 in term 1.
I20260924 04:45:16.498029 17859 raft_consensus.cc:2454] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83a072f08014cb5bc6ee55ac0f4ab84 in term 1.
I20260924 04:45:16.498378 17525 leader_election.cc:302] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888095a5485741368825e53d459766ac, e83a072f08014cb5bc6ee55ac0f4ab84; no voters:
I20260924 04:45:16.498606 17928 raft_consensus.cc:2781] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:16.498896 17928 raft_consensus.cc:686] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 1 LEADER]: Becoming Leader. State: Replica: e83a072f08014cb5bc6ee55ac0f4ab84, State: Running, Role: LEADER
I20260924 04:45:16.499315 17928 consensus_queue.cc:237] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } }
I20260924 04:45:16.502581 17446 catalog_manager.cc:5495] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 reported cstate change: term changed from 0 to 1, leader changed from <none> to e83a072f08014cb5bc6ee55ac0f4ab84 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e83a072f08014cb5bc6ee55ac0f4ab84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44691 } health_report { overall_health: HEALTHY } } }
I20260924 04:45:16.544337 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.540472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.0.98.1:44691)
0924 04:45:16.540636 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.540643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.540659 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:45:16.540753 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.540756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.540810 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.540941 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:16.541212 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.541786 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.542728 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.542956 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.543072 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.543440 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:16.543904 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:16.544077 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.544114 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.544173 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:45:16.551301 17725 raft_consensus.cc:1261] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 1 FOLLOWER]: Refusing update from remote peer e83a072f08014cb5bc6ee55ac0f4ab84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:16.551337 17859 raft_consensus.cc:1261] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 1 FOLLOWER]: Refusing update from remote peer e83a072f08014cb5bc6ee55ac0f4ab84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:16.552170 17928 consensus_queue.cc:1035] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888095a5485741368825e53d459766ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:16.552510 17933 consensus_queue.cc:1035] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486c78f3f9334a629ff31637a50a87eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:16.561071 17937 mvcc.cc:205] Tried to move back new op lower bound from 7332762077385363456 to 7332762077184004096. Current Snapshot: MvccSnapshot[applied={T|T < 7332762077385363456}]
I20260924 04:45:16.562294 17938 mvcc.cc:205] Tried to move back new op lower bound from 7332762077385363456 to 7332762077184004096. Current Snapshot: MvccSnapshot[applied={T|T < 7332762077385363456}]
I20260924 04:45:16.569126 17443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:16.569319 17443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:16.572111 17443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1890ddd15a44dd2ab94efb904650fcd: Sending DeleteTablet for 3 replicas of tablet 514902199ffc414582f93c84c2c1500a
I20260924 04:45:16.572947 17839 tablet_service.cc:1514] Processing DeleteTablet for tablet 514902199ffc414582f93c84c2c1500a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59112
I20260924 04:45:16.573031 17705 tablet_service.cc:1514] Processing DeleteTablet for tablet 514902199ffc414582f93c84c2c1500a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43812
I20260924 04:45:16.572999 17571 tablet_service.cc:1514] Processing DeleteTablet for tablet 514902199ffc414582f93c84c2c1500a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42190
I20260924 04:45:16.573524 17947 tablet_replica.cc:330] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: stopping tablet replica
I20260924 04:45:16.573657 17948 tablet_replica.cc:330] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84: stopping tablet replica
I20260924 04:45:16.573879 17946 tablet_replica.cc:330] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: stopping tablet replica
I20260924 04:45:16.574039 17948 raft_consensus.cc:2229] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:45:16.574259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17508
I20260924 04:45:16.574275 17946 raft_consensus.cc:2229] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:16.574452 17948 raft_consensus.cc:2258] T 514902199ffc414582f93c84c2c1500a P e83a072f08014cb5bc6ee55ac0f4ab84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:16.574584 17946 raft_consensus.cc:2258] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:16.576246 17947 raft_consensus.cc:2229] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:16.576578 17947 raft_consensus.cc:2258] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:16.577301 17946 ts_tablet_manager.cc:1905] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:16.577804 17947 ts_tablet_manager.cc:1905] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:16.580076 17947 ts_tablet_manager.cc:1918] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:16.580190 17947 log.cc:1196] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/514902199ffc414582f93c84c2c1500a
I20260924 04:45:16.580510 17947 ts_tablet_manager.cc:1939] T 514902199ffc414582f93c84c2c1500a P 888095a5485741368825e53d459766ac: Deleting consensus metadata
I20260924 04:45:16.580794 17946 ts_tablet_manager.cc:1918] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:16.580899 17946 log.cc:1196] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/514902199ffc414582f93c84c2c1500a
I20260924 04:45:16.581260 17946 ts_tablet_manager.cc:1939] T 514902199ffc414582f93c84c2c1500a P 486c78f3f9334a629ff31637a50a87eb: Deleting consensus metadata
I20260924 04:45:16.581957 17431 catalog_manager.cc:4841] TS 888095a5485741368825e53d459766ac (127.0.98.2:41375): tablet 514902199ffc414582f93c84c2c1500a (table test-table [id=f916db570b2749e6a5349624db68e56d]) successfully deleted
I20260924 04:45:16.582413 17431 catalog_manager.cc:4841] TS 486c78f3f9334a629ff31637a50a87eb (127.0.98.3:43209): tablet 514902199ffc414582f93c84c2c1500a (table test-table [id=f916db570b2749e6a5349624db68e56d]) successfully deleted
W20260924 04:45:16.582887 17433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44691 (error 108)
I20260924 04:45:16.583191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17642
I20260924 04:45:16.585295 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.584907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44691 (local address 127.0.0.1:42194)
0924 04:45:16.585095 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:16.585185 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 04:45:16.585467 17433 catalog_manager.cc:4584] TS e83a072f08014cb5bc6ee55ac0f4ab84 (127.0.98.1:44691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 514902199ffc414582f93c84c2c1500a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44691: connect: Connection refused (error 111)
I20260924 04:45:16.589576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17776
I20260924 04:45:16.596035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17414
2026-09-24T04:45:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2630 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T04:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:16Z Disabled control of system clock
I20260924 04:45:16.628456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:45:16.755431 17953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:16.755716 17953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:16.755754 17953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:16.760073 17953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:16.760149 17953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:16.760164 17953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:16.760183 17953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:16.760200 17953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:16.765138 17953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:16.766224 17953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:16.767607 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:45:16.776500 17953 server_base.cc:1034] running on GCE node
W20260924 04:45:16.776508 17958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:16.776471 17959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:16.776525 17961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:16.777360 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:16.778470 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:16.779639 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790225116779608 us; error 47 us; skew 500 ppm
I20260924 04:45:16.788110 17953 webserver.cc:460] Webserver started at http://127.0.98.62:44857/ using document root <none> and password file <none>
I20260924 04:45:16.788719 17953 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:16.788784 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:16.789011 17953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:16.791292 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0"
format_stamp: "Formatted at 2026-09-24 04:45:16 on dist-test-slave-npg2"
server_key: "dc5d3bb90f86de720f040bac65683498"
server_key_iv: "c2d3e7f46f7f7f3088b468c8a6b11309"
server_key_version: "encryptionkey@0"
I20260924 04:45:16.791836 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0"
format_stamp: "Formatted at 2026-09-24 04:45:16 on dist-test-slave-npg2"
server_key: "dc5d3bb90f86de720f040bac65683498"
server_key_iv: "c2d3e7f46f7f7f3088b468c8a6b11309"
server_key_version: "encryptionkey@0"
I20260924 04:45:16.795893 17953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:45:16.798511 17967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:16.800007 17953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:16.800119 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0"
format_stamp: "Formatted at 2026-09-24 04:45:16 on dist-test-slave-npg2"
server_key: "dc5d3bb90f86de720f040bac65683498"
server_key_iv: "c2d3e7f46f7f7f3088b468c8a6b11309"
server_key_version: "encryptionkey@0"
I20260924 04:45:16.800213 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:16.823266 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:16.824009 17953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:16.824169 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:16.831328 17953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39489
I20260924 04:45:16.831398 18019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39489 every 8 connection(s)
I20260924 04:45:16.832334 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:16.835387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 17953
I20260924 04:45:16.835537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:16.835845 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:16.835867 392 external_mini_cluster.cc:1428] Setting key f677119325acf458252e21864f421eb2
I20260924 04:45:16.842964 18020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0: Bootstrap starting.
I20260924 04:45:16.845650 18020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:16.846122 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:16.837399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.62:39489)
0924 04:45:16.838088 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:16.838104 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:16.838144 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:45:16.838983 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:16.839004 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:16.839540 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:16.839868 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:16.840257 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.841256 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.842493 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:16.842984 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:16.843180 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:16.844036 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:16.844041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:16.844054 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:16.844467 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:16.844525 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:16.844546 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:16.844634 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:16.844850 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 04:45:16.845192 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":170,"threads_started":1}
I20260924 04:45:16.846522 18020 log.cc:824] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:16.848691 18020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0: No bootstrap required, opened a new log
I20260924 04:45:16.852592 18020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39489 } }
I20260924 04:45:16.852850 18020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:16.852932 18020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ca3d4d0ba54c9c8133c6bcdbe4d9a0, State: Initialized, Role: FOLLOWER
I20260924 04:45:16.853363 18020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39489 } }
I20260924 04:45:16.853550 18020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:16.853631 18020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:16.853751 18020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:16.854717 18020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39489 } }
I20260924 04:45:16.855067 18020 leader_election.cc:302] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ca3d4d0ba54c9c8133c6bcdbe4d9a0; no voters:
I20260924 04:45:16.855422 18020 leader_election.cc:288] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:16.855620 18025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:16.855861 18025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [term 1 LEADER]: Becoming Leader. State: Replica: 38ca3d4d0ba54c9c8133c6bcdbe4d9a0, State: Running, Role: LEADER
I20260924 04:45:16.856216 18025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39489 } }
I20260924 04:45:16.856745 18020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:16.857903 18026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39489 } } }
I20260924 04:45:16.858028 18026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:16.858428 18034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:16.858397 18027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ca3d4d0ba54c9c8133c6bcdbe4d9a0. Latest consensus state: current_term: 1 leader_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ca3d4d0ba54c9c8133c6bcdbe4d9a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39489 } } }
I20260924 04:45:16.858575 18027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:16.861109 18034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:16.866822 18034 catalog_manager.cc:1347] Generated new cluster ID: eac1d7411c0e4b049acb96e9b73f541b
I20260924 04:45:16.866899 18034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:16.874334 18034 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:16.875386 18034 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:16.881502 18034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0: Generated new TSK 0
I20260924 04:45:16.882230 18034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:16.890350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39489
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:45:17.008603 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:17.008906 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:17.008948 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:17.013296 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:17.013377 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:17.013533 18044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:45:17.018668 18044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:17.019936 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:17.021368 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:45:17.031522 18044 server_base.cc:1034] running on GCE node
W20260924 04:45:17.031574 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:17.031479 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:17.031498 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:17.032250 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:17.032944 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:17.034134 18044 hybrid_clock.cc:648] HybridClock initialized: now 1790225117034122 us; error 28 us; skew 500 ppm
I20260924 04:45:17.036123 18044 webserver.cc:460] Webserver started at http://127.0.98.1:35023/ using document root <none> and password file <none>
I20260924 04:45:17.036757 18044 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:17.036851 18044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:17.037096 18044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:17.039353 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "ccc1ebc19cee43f6bc40a370003b4b3a"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "a6dd9ae8eb68f20e3e6be46d6a4c9323"
server_key_iv: "3676642b457b797ca46479805ed564b4"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.039944 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc1ebc19cee43f6bc40a370003b4b3a"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "a6dd9ae8eb68f20e3e6be46d6a4c9323"
server_key_iv: "3676642b457b797ca46479805ed564b4"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.043974 18044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:45:17.046795 18058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.048241 18044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:17.048385 18044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "ccc1ebc19cee43f6bc40a370003b4b3a"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "a6dd9ae8eb68f20e3e6be46d6a4c9323"
server_key_iv: "3676642b457b797ca46479805ed564b4"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.048509 18044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:17.068609 18044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:17.069348 18044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:17.069554 18044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:17.070194 18044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:17.071213 18044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:17.071282 18044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.071357 18044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:17.071411 18044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.081060 18044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45621
I20260924 04:45:17.081130 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45621 every 8 connection(s)
I20260924 04:45:17.082048 18044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:45:17.087674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18044
I20260924 04:45:17.087824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:45:17.088176 392 external_mini_cluster.cc:1428] Setting key 8cf7b0c2c142d8241441ce474066b909
I20260924 04:45:17.090268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39489
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:45:17.092523 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.084182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44499 (local address 127.0.98.62:39489)
0924 04:45:17.084329 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.084333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.085482 (+ 1149us) server_negotiation.cc:407] Connection header received
0924 04:45:17.086861 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.086864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.086912 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.086994 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:17.088036 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.088792 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.090162 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.090487 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.090716 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.091951 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:17.091958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:17.091963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:17.091979 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:17.092005 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:17.092016 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:17.092099 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.092253 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.092338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:45:17.093605 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.084606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39489 (local address 127.0.98.1:44499)
0924 04:45:17.085319 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.085361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.086646 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.087215 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.087223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.087309 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:17.087851 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.087871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.088925 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.088928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.090027 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.090037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.090761 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.090765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.090943 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.091668 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:17.091670 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:17.091714 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:17.091784 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:17.092108 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:17.092114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:17.092119 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.092221 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.092454 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260924 04:45:17.094933 18172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39489
I20260924 04:45:17.095258 18172 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:17.095782 18172 heartbeater.cc:511] Master 127.0.98.62:39489 requested a full tablet report, sending...
I20260924 04:45:17.097380 17984 ts_manager.cc:194] Registered new tserver with Master: ccc1ebc19cee43f6bc40a370003b4b3a (127.0.98.1:45621)
I20260924 04:45:17.098569 17984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44499
W20260924 04:45:17.206173 18175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:17.206450 18175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:17.206491 18175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:17.210731 18175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:17.210803 18175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:17.210888 18175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:45:17.215947 18175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:17.217103 18175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:17.218451 18175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:17.228350 18180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:17.228376 18181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:17.228374 18183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:17.228508 18175 server_base.cc:1034] running on GCE node
I20260924 04:45:17.229048 18175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:17.229803 18175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:17.230985 18175 hybrid_clock.cc:648] HybridClock initialized: now 1790225117230969 us; error 35 us; skew 500 ppm
I20260924 04:45:17.232874 18175 webserver.cc:460] Webserver started at http://127.0.98.2:45587/ using document root <none> and password file <none>
I20260924 04:45:17.233489 18175 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:17.233594 18175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:17.233837 18175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:17.235987 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "5adc1064fdae44168ca18e0d4e49b7cd"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "b265000a8b138d33558aa389add0cd00"
server_key_iv: "0755346870cdf01bcd8745c2e0cfbfd9"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.236624 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5adc1064fdae44168ca18e0d4e49b7cd"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "b265000a8b138d33558aa389add0cd00"
server_key_iv: "0755346870cdf01bcd8745c2e0cfbfd9"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.240363 18175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:45:17.242913 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.244243 18175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:17.244391 18175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "5adc1064fdae44168ca18e0d4e49b7cd"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "b265000a8b138d33558aa389add0cd00"
server_key_iv: "0755346870cdf01bcd8745c2e0cfbfd9"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.244517 18175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:17.269440 18175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:17.270241 18175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:17.270458 18175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:17.271085 18175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:17.272119 18175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:17.272207 18175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.272279 18175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:17.272331 18175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.281519 18175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42219
I20260924 04:45:17.281538 18302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42219 every 8 connection(s)
I20260924 04:45:17.282485 18175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:45:17.287636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18175
I20260924 04:45:17.287796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:45:17.288081 392 external_mini_cluster.cc:1428] Setting key 984f2a20a139a7197fa089a387fae72a
I20260924 04:45:17.290066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39489
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:45:17.292716 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.284589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45041 (local address 127.0.98.62:39489)
0924 04:45:17.284736 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.284740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.285747 (+ 1007us) server_negotiation.cc:407] Connection header received
0924 04:45:17.287246 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.287249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.287305 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.287415 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:17.288509 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.289154 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.290525 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.290936 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.291066 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.292211 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:17.292215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:17.292218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:17.292238 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:17.292257 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:17.292265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:17.292349 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.292511 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.292574 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:45:17.293936 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.284906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39489 (local address 127.0.98.2:45041)
0924 04:45:17.285591 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.285631 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.286994 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.287623 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.287630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.287765 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:17.288371 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.288385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.289334 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.289340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.290335 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.290348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.291095 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.291098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.291300 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.291940 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:17.291942 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:17.291983 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:17.292048 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:17.292352 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:17.292363 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:17.292370 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.292511 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.292811 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260924 04:45:17.295256 18303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39489
I20260924 04:45:17.295526 18303 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:17.296052 18303 heartbeater.cc:511] Master 127.0.98.62:39489 requested a full tablet report, sending...
I20260924 04:45:17.297142 17984 ts_manager.cc:194] Registered new tserver with Master: 5adc1064fdae44168ca18e0d4e49b7cd (127.0.98.2:42219)
I20260924 04:45:17.298055 17984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45041
W20260924 04:45:17.404157 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:17.404409 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:17.404448 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:17.408447 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:17.408506 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:17.408565 18306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:45:17.413341 18306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:17.414439 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:17.415895 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:17.426800 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:17.426885 18306 server_base.cc:1034] running on GCE node
W20260924 04:45:17.426800 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:17.426829 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:17.427441 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:17.428164 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:17.429359 18306 hybrid_clock.cc:648] HybridClock initialized: now 1790225117429344 us; error 28 us; skew 500 ppm
I20260924 04:45:17.431358 18306 webserver.cc:460] Webserver started at http://127.0.98.3:45059/ using document root <none> and password file <none>
I20260924 04:45:17.431990 18306 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:17.432056 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:17.432307 18306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:17.434391 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ad043918a134125b70dfce5c4fdf6eb"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "9c7414c46572815110ff4f8011ae820c"
server_key_iv: "711099644ab67b42bac7b50f8ce2c11b"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.434960 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ad043918a134125b70dfce5c4fdf6eb"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "9c7414c46572815110ff4f8011ae820c"
server_key_iv: "711099644ab67b42bac7b50f8ce2c11b"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.438773 18306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:45:17.441380 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.442734 18306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:17.442898 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "6ad043918a134125b70dfce5c4fdf6eb"
format_stamp: "Formatted at 2026-09-24 04:45:17 on dist-test-slave-npg2"
server_key: "9c7414c46572815110ff4f8011ae820c"
server_key_iv: "711099644ab67b42bac7b50f8ce2c11b"
server_key_version: "encryptionkey@0"
I20260924 04:45:17.443024 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:17.463842 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:17.464592 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:17.464797 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:17.465411 18306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:17.466377 18306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:17.466460 18306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.466535 18306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:17.466569 18306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:17.476209 18306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44779
I20260924 04:45:17.476233 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44779 every 8 connection(s)
I20260924 04:45:17.477159 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:45:17.478006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18306
I20260924 04:45:17.478137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:45:17.478494 392 external_mini_cluster.cc:1428] Setting key b65e3eee4f58ab7b3ad565aa3b84a826
I20260924 04:45:17.486966 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.479334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33839 (local address 127.0.98.62:39489)
0924 04:45:17.479490 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.479495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.480719 (+ 1224us) server_negotiation.cc:407] Connection header received
0924 04:45:17.482198 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.482201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.482255 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.482335 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:17.483276 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.483838 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.484986 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.485218 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.485332 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.486442 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:17.486452 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:17.486454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:17.486466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:17.486484 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:17.486492 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:17.486637 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.486750 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.486818 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:45:17.487826 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.479809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39489 (local address 127.0.98.3:33839)
0924 04:45:17.480577 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.480637 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.481976 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.482514 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.482521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.482611 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:17.483109 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.483144 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.483970 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.483974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.484860 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.484869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.485372 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.485376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.485542 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.486178 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:17.486180 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:17.486221 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:17.486286 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:17.486585 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:17.486594 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:17.486600 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.486752 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.487039 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":164,"threads_started":1}
I20260924 04:45:17.488988 18434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39489
I20260924 04:45:17.489193 18434 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:17.489703 18434 heartbeater.cc:511] Master 127.0.98.62:39489 requested a full tablet report, sending...
I20260924 04:45:17.490741 17984 ts_manager.cc:194] Registered new tserver with Master: 6ad043918a134125b70dfce5c4fdf6eb (127.0.98.3:44779)
I20260924 04:45:17.491406 17984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33839
I20260924 04:45:17.492740 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:45:17.498885 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.494856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.0.98.62:39489)
0924 04:45:17.495041 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.495046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.495274 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:45:17.495439 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.495442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.495494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.495590 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:17.495878 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.496406 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.497274 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.497499 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.497618 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.498455 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:17.498459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:17.498462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:17.498474 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:17.498494 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:17.498511 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:17.498594 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.498684 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.498740 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 04:45:17.501705 17983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:45:17.504941 17983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:45:17.520541 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.514081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44779 (local address 127.0.0.1:52302)
0924 04:45:17.514722 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.514775 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.514923 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.516025 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.516032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.516063 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:17.516210 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.516222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.517862 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.517865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.519014 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.519022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.520076 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.520080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.520185 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.520218 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.520281 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.520352 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":26,"spinlock_wait_cycles":3456,"thread_start_us":144,"threads_started":1}
I20260924 04:45:17.520613 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.515216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.98.3:44779)
0924 04:45:17.515563 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.515577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.515602 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:45:17.515680 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.515685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.515869 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.516022 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:17.516336 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.517601 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.519169 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.519946 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.520066 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.520245 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.520316 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.520436 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
I20260924 04:45:17.520784 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.514769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.98.1:45621)
0924 04:45:17.515178 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.515191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.515548 (+ 357us) server_negotiation.cc:407] Connection header received
0924 04:45:17.515631 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.515637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.515861 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.516012 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:17.516469 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.517197 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.518800 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.519646 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.519755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.519952 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.520500 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.520610 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":158,"threads_started":1}
I20260924 04:45:17.520380 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.514348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45621 (local address 127.0.0.1:50140)
0924 04:45:17.515341 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.515356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.515447 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.516183 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.516185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.516205 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:17.516308 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.516314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.517314 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.517321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.518663 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.518673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.519785 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.519788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.519903 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.519949 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.520066 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.520198 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260924 04:45:17.521102 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.514145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.0.98.2:42219)
0924 04:45:17.515652 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.515663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.515684 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:45:17.515810 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.515815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.515975 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.516100 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:17.516430 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.517598 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.519352 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.520134 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.520237 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.520419 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.520815 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.520925 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":133,"threads_started":1}
I20260924 04:45:17.521674 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.513893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42219 (local address 127.0.0.1:35256)
0924 04:45:17.514723 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.514775 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.515673 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.516232 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.516235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.516247 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:17.516314 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.516319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.517768 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.517771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.519125 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.519191 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.520449 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.520454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.520535 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.520554 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.520644 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.521548 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":252,"spinlock_wait_cycles":6016,"thread_start_us":80,"threads_started":1}
I20260924 04:45:17.524001 18368 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcc063308a2463a99a559b4c4d0b86b (DEFAULT_TABLE table=test-table [id=8c4faec3a044498dbeac1afe88cb7df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:17.524121 18106 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcc063308a2463a99a559b4c4d0b86b (DEFAULT_TABLE table=test-table [id=8c4faec3a044498dbeac1afe88cb7df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:17.524442 18237 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcc063308a2463a99a559b4c4d0b86b (DEFAULT_TABLE table=test-table [id=8c4faec3a044498dbeac1afe88cb7df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:17.525094 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcc063308a2463a99a559b4c4d0b86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:17.525178 18106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcc063308a2463a99a559b4c4d0b86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:17.525363 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcc063308a2463a99a559b4c4d0b86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:17.531257 18451 tablet_bootstrap.cc:492] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Bootstrap starting.
I20260924 04:45:17.532692 18452 tablet_bootstrap.cc:492] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Bootstrap starting.
I20260924 04:45:17.533439 18453 tablet_bootstrap.cc:492] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: Bootstrap starting.
I20260924 04:45:17.534597 18451 tablet_bootstrap.cc:654] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:17.534883 18452 tablet_bootstrap.cc:654] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:17.535305 18453 tablet_bootstrap.cc:654] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:17.535538 18451 log.cc:824] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:17.535669 18452 log.cc:824] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:17.536186 18453 log.cc:824] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:17.537523 18452 tablet_bootstrap.cc:492] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: No bootstrap required, opened a new log
I20260924 04:45:17.537603 18451 tablet_bootstrap.cc:492] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: No bootstrap required, opened a new log
I20260924 04:45:17.537750 18452 ts_tablet_manager.cc:1397] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:45:17.537791 18451 ts_tablet_manager.cc:1397] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:45:17.538080 18453 tablet_bootstrap.cc:492] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: No bootstrap required, opened a new log
I20260924 04:45:17.538254 18453 ts_tablet_manager.cc:1397] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:45:17.541460 18452 raft_consensus.cc:348] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.541738 18452 raft_consensus.cc:374] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:17.541817 18452 raft_consensus.cc:729] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5adc1064fdae44168ca18e0d4e49b7cd, State: Initialized, Role: FOLLOWER
I20260924 04:45:17.542351 18452 consensus_queue.cc:260] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.542331 18451 raft_consensus.cc:348] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.542465 18453 raft_consensus.cc:348] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.542649 18451 raft_consensus.cc:374] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:17.542695 18453 raft_consensus.cc:374] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:17.542702 18451 raft_consensus.cc:729] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad043918a134125b70dfce5c4fdf6eb, State: Initialized, Role: FOLLOWER
I20260924 04:45:17.542833 18453 raft_consensus.cc:729] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc1ebc19cee43f6bc40a370003b4b3a, State: Initialized, Role: FOLLOWER
I20260924 04:45:17.543102 18303 heartbeater.cc:503] Master 127.0.98.62:39489 was elected leader, sending a full tablet report...
I20260924 04:45:17.543232 18451 consensus_queue.cc:260] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.543301 18453 consensus_queue.cc:260] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.543460 18452 ts_tablet_manager.cc:1428] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:45:17.544162 18453 ts_tablet_manager.cc:1428] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:45:17.544162 18451 ts_tablet_manager.cc:1428] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 04:45:17.544193 18172 heartbeater.cc:503] Master 127.0.98.62:39489 was elected leader, sending a full tablet report...
I20260924 04:45:17.544497 18434 heartbeater.cc:503] Master 127.0.98.62:39489 was elected leader, sending a full tablet report...
W20260924 04:45:17.583371 18173 tablet.cc:2369] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:17.687172 18459 raft_consensus.cc:482] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:17.687422 18459 raft_consensus.cc:504] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.688757 18459 leader_election.cc:288] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5adc1064fdae44168ca18e0d4e49b7cd (127.0.98.2:42219), 6ad043918a134125b70dfce5c4fdf6eb (127.0.98.3:44779)
I20260924 04:45:17.694299 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.689238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49335 (local address 127.0.98.3:44779)
0924 04:45:17.689356 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.689362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.689652 (+ 290us) server_negotiation.cc:407] Connection header received
0924 04:45:17.689715 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.689721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.689770 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.689844 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:17.690154 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.691005 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.692879 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.693577 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.693694 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.693745 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.694093 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.694181 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:45:17.694299 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.689123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56765 (local address 127.0.98.2:42219)
0924 04:45:17.689330 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.689335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.689432 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:45:17.689579 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.689582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.689660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.689750 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:17.690154 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.690964 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.692879 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.693533 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.693657 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.693728 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.694093 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.694171 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 04:45:17.694329 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.689103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44779 (local address 127.0.98.1:49335)
0924 04:45:17.689548 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.689565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.689648 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.689875 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.689878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.689890 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:17.689983 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.689989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.691180 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.691184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.692730 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.692741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.693695 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.693699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.693898 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.693973 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.694060 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.694150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":262,"thread_start_us":83,"threads_started":1}
I20260924 04:45:17.694329 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.688942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42219 (local address 127.0.98.1:56765)
0924 04:45:17.689320 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.689339 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:17.689464 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:17.689783 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:17.689785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:17.689798 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:17.689985 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.689990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.691180 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.691184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.692730 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:17.692741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.693695 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.693699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:17.693936 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.693973 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:45:17.694060 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:45:17.694172 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":209,"thread_start_us":142,"threads_started":1}
I20260924 04:45:17.695060 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcc063308a2463a99a559b4c4d0b86b" candidate_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad043918a134125b70dfce5c4fdf6eb" is_pre_election: true
I20260924 04:45:17.695153 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcc063308a2463a99a559b4c4d0b86b" candidate_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" is_pre_election: true
I20260924 04:45:17.695432 18257 raft_consensus.cc:2454] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc1ebc19cee43f6bc40a370003b4b3a in term 0.
I20260924 04:45:17.695441 18388 raft_consensus.cc:2454] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc1ebc19cee43f6bc40a370003b4b3a in term 0.
I20260924 04:45:17.695969 18062 leader_election.cc:302] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad043918a134125b70dfce5c4fdf6eb, ccc1ebc19cee43f6bc40a370003b4b3a; no voters:
I20260924 04:45:17.696233 18459 raft_consensus.cc:2781] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:45:17.696323 18459 raft_consensus.cc:482] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:45:17.696367 18459 raft_consensus.cc:3037] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:17.697553 18459 raft_consensus.cc:504] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.697942 18459 leader_election.cc:288] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5adc1064fdae44168ca18e0d4e49b7cd (127.0.98.2:42219), 6ad043918a134125b70dfce5c4fdf6eb (127.0.98.3:44779)
I20260924 04:45:17.698290 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcc063308a2463a99a559b4c4d0b86b" candidate_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5adc1064fdae44168ca18e0d4e49b7cd"
I20260924 04:45:17.698396 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcc063308a2463a99a559b4c4d0b86b" candidate_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad043918a134125b70dfce5c4fdf6eb"
I20260924 04:45:17.698417 18257 raft_consensus.cc:3037] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:17.698540 18388 raft_consensus.cc:3037] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:17.699613 18257 raft_consensus.cc:2454] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc1ebc19cee43f6bc40a370003b4b3a in term 1.
I20260924 04:45:17.699774 18388 raft_consensus.cc:2454] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc1ebc19cee43f6bc40a370003b4b3a in term 1.
I20260924 04:45:17.699980 18060 leader_election.cc:302] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5adc1064fdae44168ca18e0d4e49b7cd, ccc1ebc19cee43f6bc40a370003b4b3a; no voters:
I20260924 04:45:17.700168 18459 raft_consensus.cc:2781] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:17.700377 18459 raft_consensus.cc:686] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [term 1 LEADER]: Becoming Leader. State: Replica: ccc1ebc19cee43f6bc40a370003b4b3a, State: Running, Role: LEADER
I20260924 04:45:17.700762 18459 consensus_queue.cc:237] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } }
I20260924 04:45:17.704595 17983 catalog_manager.cc:5495] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to ccc1ebc19cee43f6bc40a370003b4b3a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccc1ebc19cee43f6bc40a370003b4b3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45621 } health_report { overall_health: HEALTHY } } }
W20260924 04:45:17.728246 18435 tablet.cc:2369] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:17.759299 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.755638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.0.98.1:45621)
0924 04:45:17.755765 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:17.755769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:17.755786 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:45:17.755957 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:17.755959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:17.756019 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:17.756141 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:17.756379 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.756929 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.757815 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:17.758096 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:17.758177 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:17.758438 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:17.758857 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:17.758974 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:17.759061 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:45:17.759184 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:45:17.765195 18257 raft_consensus.cc:1261] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 1 FOLLOWER]: Refusing update from remote peer ccc1ebc19cee43f6bc40a370003b4b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:17.765194 18388 raft_consensus.cc:1261] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 1 FOLLOWER]: Refusing update from remote peer ccc1ebc19cee43f6bc40a370003b4b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:17.765926 18459 consensus_queue.cc:1035] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5adc1064fdae44168ca18e0d4e49b7cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:17.766180 18462 consensus_queue.cc:1035] T 9bcc063308a2463a99a559b4c4d0b86b P ccc1ebc19cee43f6bc40a370003b4b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad043918a134125b70dfce5c4fdf6eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:17.774605 18466 mvcc.cc:205] Tried to move back new op lower bound from 7332762082359021568 to 7332762082105421824. Current Snapshot: MvccSnapshot[applied={T|T < 7332762082359021568}]
I20260924 04:45:17.784796 17983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:17.784961 17983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:17.787596 17983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38ca3d4d0ba54c9c8133c6bcdbe4d9a0: Sending DeleteTablet for 3 replicas of tablet 9bcc063308a2463a99a559b4c4d0b86b
I20260924 04:45:17.788513 18237 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcc063308a2463a99a559b4c4d0b86b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:17 UTC) from {username='slave'} at 127.0.0.1:35256
I20260924 04:45:17.788545 18106 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcc063308a2463a99a559b4c4d0b86b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:17 UTC) from {username='slave'} at 127.0.0.1:50140
I20260924 04:45:17.788694 18368 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcc063308a2463a99a559b4c4d0b86b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:17 UTC) from {username='slave'} at 127.0.0.1:52302
I20260924 04:45:17.789129 18476 tablet_replica.cc:330] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: stopping tablet replica
I20260924 04:45:17.789258 18477 tablet_replica.cc:330] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: stopping tablet replica
I20260924 04:45:17.789515 18477 raft_consensus.cc:2229] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:17.789515 18476 raft_consensus.cc:2229] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:17.789788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18044
I20260924 04:45:17.789911 18476 raft_consensus.cc:2258] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:17.789911 18477 raft_consensus.cc:2258] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:17.791983 18476 ts_tablet_manager.cc:1905] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:17.792361 18477 ts_tablet_manager.cc:1905] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:17.795042 18477 ts_tablet_manager.cc:1918] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:17.795207 18477 log.cc:1196] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/9bcc063308a2463a99a559b4c4d0b86b
I20260924 04:45:17.795517 18477 ts_tablet_manager.cc:1939] T 9bcc063308a2463a99a559b4c4d0b86b P 6ad043918a134125b70dfce5c4fdf6eb: Deleting consensus metadata
I20260924 04:45:17.795701 18476 ts_tablet_manager.cc:1918] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:17.795816 18476 log.cc:1196] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/9bcc063308a2463a99a559b4c4d0b86b
I20260924 04:45:17.796154 18476 ts_tablet_manager.cc:1939] T 9bcc063308a2463a99a559b4c4d0b86b P 5adc1064fdae44168ca18e0d4e49b7cd: Deleting consensus metadata
I20260924 04:45:17.796972 17971 catalog_manager.cc:4841] TS 6ad043918a134125b70dfce5c4fdf6eb (127.0.98.3:44779): tablet 9bcc063308a2463a99a559b4c4d0b86b (table test-table [id=8c4faec3a044498dbeac1afe88cb7df0]) successfully deleted
I20260924 04:45:17.797528 17969 catalog_manager.cc:4841] TS 5adc1064fdae44168ca18e0d4e49b7cd (127.0.98.2:42219): tablet 9bcc063308a2463a99a559b4c4d0b86b (table test-table [id=8c4faec3a044498dbeac1afe88cb7df0]) successfully deleted
W20260924 04:45:17.798099 17968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45621 (error 108)
I20260924 04:45:17.798969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18175
I20260924 04:45:17.799297 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:17.798925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45621 (local address 127.0.0.1:50160)
0924 04:45:17.799078 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:17.799191 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:45:17.799719 17968 catalog_manager.cc:4584] TS ccc1ebc19cee43f6bc40a370003b4b3a (127.0.98.1:45621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bcc063308a2463a99a559b4c4d0b86b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45621: connect: Connection refused (error 111)
I20260924 04:45:17.805322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18306
I20260924 04:45:17.811206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 17953
2026-09-24T04:45:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:17 dist-test-slave-npg2 krb5kdc[18481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:17 dist-test-slave-npg2 krb5kdc[18481](info): set up 2 sockets
Sep 24 04:45:17 dist-test-slave-npg2 krb5kdc[18481](info): commencing operation
krb5kdc: starting...
W20260924 04:45:18.858940 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:18 dist-test-slave-npg2 krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:18.987638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:19.101284 18494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:19.101599 18494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:19.101681 18494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:19.105695 18494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:19.105793 18494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:19.105832 18494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:19.105886 18494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:19.105923 18494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:19.111197 18494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:19.112286 18494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:19.113397 18494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:19.122087 18499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:19.122073 18500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:19.122110 18502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:19.122186 18494 server_base.cc:1034] running on GCE node
I20260924 04:45:19.122995 18494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:19.124289 18494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:19.125478 18494 hybrid_clock.cc:648] HybridClock initialized: now 1790225119125484 us; error 80 us; skew 500 ppm
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:19.128579 18494 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:19.129524 18494 webserver.cc:460] Webserver started at http://127.0.98.62:34861/ using document root <none> and password file <none>
I20260924 04:45:19.130120 18494 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:19.130224 18494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:19.130482 18494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:19.132661 18494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "e7f6820a9f7e4f039c33650fa1c660b6"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "ee29fe1105ba5c0823562a7f7f61ce02"
server_key_iv: "766de5d30f04b4383445755652c569a0"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.133296 18494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7f6820a9f7e4f039c33650fa1c660b6"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "ee29fe1105ba5c0823562a7f7f61ce02"
server_key_iv: "766de5d30f04b4383445755652c569a0"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.137488 18494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:19.140868 18509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.142418 18494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:19.142561 18494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "e7f6820a9f7e4f039c33650fa1c660b6"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "ee29fe1105ba5c0823562a7f7f61ce02"
server_key_iv: "766de5d30f04b4383445755652c569a0"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.142689 18494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:19.170203 18494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:19.173316 18494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:19.173512 18494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:19.180441 18494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39863
I20260924 04:45:19.180464 18561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39863 every 8 connection(s)
I20260924 04:45:19.181416 18494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:19.184150 18562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:19.184641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18494
I20260924 04:45:19.184778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:19.185082 392 external_mini_cluster.cc:1428] Setting key c403d43b2f907622097c0055554be428
I20260924 04:45:19.191201 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6: Bootstrap starting.
I20260924 04:45:19.194088 18562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6: Neither blocks nor log segments found. Creating new log.
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:19.195027 18562 log.cc:824] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:19.197263 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6: No bootstrap required, opened a new log
I20260924 04:45:19.200425 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.186444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.0.98.62:39863)
0924 04:45:19.186882 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.186898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.186933 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:45:19.187668 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.187686 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.188008 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.188259 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:19.189589 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.190600 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.191852 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.192325 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.192445 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.195552 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:19.195559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:19.195632 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:19.195668 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:19.197455 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.197937 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.197941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.197945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.198141 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.198442 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.198444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.198445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.198873 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:19.199033 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.199362 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.199697 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":118,"threads_started":1}
I20260924 04:45:19.201382 18562 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 04:45:19.201606 18562 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:19.201678 18562 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f6820a9f7e4f039c33650fa1c660b6, State: Initialized, Role: FOLLOWER
I20260924 04:45:19.202090 18562 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 04:45:19.202204 18562 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:19.202291 18562 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:19.202425 18562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:19.203438 18562 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 04:45:19.203787 18562 leader_election.cc:302] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7f6820a9f7e4f039c33650fa1c660b6; no voters:
I20260924 04:45:19.204129 18562 leader_election.cc:288] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:19.204209 18567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:19.204423 18567 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [term 1 LEADER]: Becoming Leader. State: Replica: e7f6820a9f7e4f039c33650fa1c660b6, State: Running, Role: LEADER
I20260924 04:45:19.204777 18567 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 04:45:19.205430 18562 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:19.206459 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260924 04:45:19.206593 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:19.206884 18569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7f6820a9f7e4f039c33650fa1c660b6. Latest consensus state: current_term: 1 leader_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f6820a9f7e4f039c33650fa1c660b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260924 04:45:19.206964 18569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:19.206986 18576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:19.209729 18576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:19.215961 18576 catalog_manager.cc:1347] Generated new cluster ID: 0915efe30efa4982a44a2e062ebe56d8
I20260924 04:45:19.216055 18576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:19.235692 18576 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:19.237203 18576 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:19.243629 18576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6: Generated new TSK 0
I20260924 04:45:19.244498 18576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:45:19.276295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:19.390709 18588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:19.391016 18588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:19.391077 18588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:19.395144 18588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:19.395251 18588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:19.395347 18588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:45:19.400715 18588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:19.401777 18588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:19.402807 18588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:19.412842 18594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:19.412866 18596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:19.412846 18593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:19.413103 18588 server_base.cc:1034] running on GCE node
I20260924 04:45:19.413789 18588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:19.414394 18588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:19.415565 18588 hybrid_clock.cc:648] HybridClock initialized: now 1790225119415542 us; error 34 us; skew 500 ppm
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:19.418401 18588 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:45:19.419303 18588 webserver.cc:460] Webserver started at http://127.0.98.1:41445/ using document root <none> and password file <none>
I20260924 04:45:19.419836 18588 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:19.419894 18588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:19.420065 18588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:19.422048 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "6ce3903ac1e84d7aad47b20b8b7933ce"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "6d8e5619bc0f48b6c0f5880ad2682448"
server_key_iv: "9bd5cb25242ba4a2ca82e530d98a8902"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.422550 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6ce3903ac1e84d7aad47b20b8b7933ce"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "6d8e5619bc0f48b6c0f5880ad2682448"
server_key_iv: "9bd5cb25242ba4a2ca82e530d98a8902"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.426587 18588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:19.430203 18603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.431561 18588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:19.431701 18588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "6ce3903ac1e84d7aad47b20b8b7933ce"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "6d8e5619bc0f48b6c0f5880ad2682448"
server_key_iv: "9bd5cb25242ba4a2ca82e530d98a8902"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.431820 18588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:19.471812 18588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:19.475539 18588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:19.475771 18588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:19.476418 18588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:19.477278 18588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:19.477324 18588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.477424 18588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:19.477456 18588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.487210 18588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42767
I20260924 04:45:19.487236 18716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42767 every 8 connection(s)
I20260924 04:45:19.488158 18588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:45:19.493575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18588
I20260924 04:45:19.493722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:45:19.494081 392 external_mini_cluster.cc:1428] Setting key 47a47c339625629ceadfa220f8420e62
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:19.503525 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.490102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42801 (local address 127.0.98.62:39863)
0924 04:45:19.490251 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.490255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.491057 (+ 802us) server_negotiation.cc:407] Connection header received
0924 04:45:19.492431 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.492435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.492497 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.492576 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:19.494033 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.494790 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.496174 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.496403 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.496519 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.499201 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:19.499208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:19.499215 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:19.499251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:19.501071 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.501696 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.501702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.501705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.501805 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.502174 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.502180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.502184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.502447 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:19.502557 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.503100 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.503285 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:45:19.504446 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.490438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:42801)
0924 04:45:19.490885 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:19.490930 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:19.492212 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:19.492752 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:19.492758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:19.493116 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:19.493832 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.493856 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.494924 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.494927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.496040 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.496050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.496567 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.496571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.496747 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.497380 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:19.497396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:19.498991 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:19.501209 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:19.501214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:19.501226 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:19.501575 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:19.501911 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:19.501914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:19.501915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:19.502065 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:19.502588 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:19.502593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:19.502841 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 04:45:19.503011 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:45:19.503475 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260924 04:45:19.505908 18717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 04:45:19.506209 18717 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:45:19.506767 18717 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 04:45:19.508635 18526 ts_manager.cc:194] Registered new tserver with Master: 6ce3903ac1e84d7aad47b20b8b7933ce (127.0.98.1:42767)
I20260924 04:45:19.510013 18526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:45:19.527218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:19.645502 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:19.645792 18722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:19.645836 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:19.650039 18722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:19.650110 18722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:19.650172 18722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:45:19.655824 18722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:19.657042 18722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:19.658089 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:19.667824 18728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:19.667856 18722 server_base.cc:1034] running on GCE node
W20260924 04:45:19.667814 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:19.667814 18730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:19.668391 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:19.668972 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:19.670154 18722 hybrid_clock.cc:648] HybridClock initialized: now 1790225119670140 us; error 26 us; skew 500 ppm
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:19.672971 18722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:45:19.673828 18722 webserver.cc:460] Webserver started at http://127.0.98.2:40789/ using document root <none> and password file <none>
I20260924 04:45:19.674388 18722 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:19.674448 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:19.674633 18722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:19.676680 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad38b166fbd949a3856473584cb64ee4"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "4ab7e953da6abec2d789f2a0182549a3"
server_key_iv: "9376f69afd979a3ef4c7ad2cb819fcef"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.677188 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad38b166fbd949a3856473584cb64ee4"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "4ab7e953da6abec2d789f2a0182549a3"
server_key_iv: "9376f69afd979a3ef4c7ad2cb819fcef"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.680871 18722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:45:19.683300 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.684504 18722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:45:19.684609 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "ad38b166fbd949a3856473584cb64ee4"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "4ab7e953da6abec2d789f2a0182549a3"
server_key_iv: "9376f69afd979a3ef4c7ad2cb819fcef"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.684703 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:19.696470 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:19.699910 18722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:19.700121 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:19.700711 18722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:19.701535 18722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:19.701606 18722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.701678 18722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:19.701722 18722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.711112 18722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39577
I20260924 04:45:19.711146 18850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39577 every 8 connection(s)
I20260924 04:45:19.712081 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:45:19.714157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18722
I20260924 04:45:19.714288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:45:19.714576 392 external_mini_cluster.cc:1428] Setting key 609dc379f04094e8fda3d88a320f6389
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:45:19.728116 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.714138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54811 (local address 127.0.98.62:39863)
0924 04:45:19.714323 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.714329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.715176 (+ 847us) server_negotiation.cc:407] Connection header received
0924 04:45:19.716669 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.716671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.716716 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.716787 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:19.718356 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.718888 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.720771 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.720995 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.721088 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.724375 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:19.724379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:19.724380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:19.724406 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:19.725909 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.726451 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.726459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.726460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.726540 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.726904 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.726907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.726909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.727104 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:19.727265 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.727798 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.727929 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:45:19.729331 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.714487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:54811)
0924 04:45:19.715002 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:19.715046 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:19.716367 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:19.717017 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:19.717028 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:19.717592 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:19.718202 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.718217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.719054 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.719061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.720638 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.720652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.721169 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.721176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.721443 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.722213 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:19.722242 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:19.724140 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:19.726040 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:19.726047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:19.726064 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:19.726340 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:19.726651 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:19.726654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:19.726656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:19.726804 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:19.727276 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:19.727285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:19.727550 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 04:45:19.727743 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:45:19.728136 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":142,"threads_started":1}
I20260924 04:45:19.730966 18851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 04:45:19.731348 18851 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:19.731964 18851 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 04:45:19.733289 18526 ts_manager.cc:194] Registered new tserver with Master: ad38b166fbd949a3856473584cb64ee4 (127.0.98.2:39577)
I20260924 04:45:19.733932 18526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:45:19.746289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:19.861577 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:19.861845 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:19.861888 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:19.865900 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:19.865959 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:19.866021 18856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:45:19.871341 18856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:19.872387 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:19.873364 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:19.883517 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:19.883522 18864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:19.883570 18856 server_base.cc:1034] running on GCE node
W20260924 04:45:19.883517 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:19.884274 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:19.884888 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:19.886046 18856 hybrid_clock.cc:648] HybridClock initialized: now 1790225119886014 us; error 46 us; skew 500 ppm
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:19.888881 18856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:45:19.889740 18856 webserver.cc:460] Webserver started at http://127.0.98.3:39073/ using document root <none> and password file <none>
I20260924 04:45:19.890307 18856 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:19.890369 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:19.890617 18856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:19.892702 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "86f28d9c5abe46d3964a969cb51649a3"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "e86837ecfda040a8426e93e9227c003f"
server_key_iv: "9b94eb981491b69478e524139246937f"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.893258 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "86f28d9c5abe46d3964a969cb51649a3"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "e86837ecfda040a8426e93e9227c003f"
server_key_iv: "9b94eb981491b69478e524139246937f"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.896958 18856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:45:19.899573 18871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.900792 18856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:45:19.900928 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "86f28d9c5abe46d3964a969cb51649a3"
format_stamp: "Formatted at 2026-09-24 04:45:19 on dist-test-slave-npg2"
server_key: "e86837ecfda040a8426e93e9227c003f"
server_key_iv: "9b94eb981491b69478e524139246937f"
server_key_version: "encryptionkey@0"
I20260924 04:45:19.901041 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:19.913048 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:19.916047 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:19.916256 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:19.916852 18856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:19.917666 18856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:19.917742 18856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.917831 18856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:19.917882 18856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:19.927505 18856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40643
I20260924 04:45:19.927534 18984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40643 every 8 connection(s)
I20260924 04:45:19.928474 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:45:19.932835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 18856
I20260924 04:45:19.933005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:45:19.933288 392 external_mini_cluster.cc:1428] Setting key c2421dc6d78a6a826844b9c308562a15
Sep 24 04:45:19 dist-test-slave-npg2 krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:19.942508 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.930429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54749 (local address 127.0.98.62:39863)
0924 04:45:19.930563 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.930628 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.931494 (+ 866us) server_negotiation.cc:407] Connection header received
0924 04:45:19.932841 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.932844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.932901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.932982 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:19.934331 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.934885 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.936195 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.936429 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.936542 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.938918 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:19.938924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:19.938932 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:19.938956 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:19.940499 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.941046 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.941048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.941050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.941138 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.941448 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.941450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.941452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.941625 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:19.941703 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.942248 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.942384 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:45:19.943601 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.930849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:54749)
0924 04:45:19.931349 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:19.931391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:19.932580 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:19.933159 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:19.933166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:19.933593 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:19.934189 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.934203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.935019 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.935024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.936064 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.936080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.936581 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.936591 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.936760 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.937345 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:19.937360 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:19.938756 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:19.940660 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:19.940665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:19.940678 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:19.940936 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:19.941224 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:19.941226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:19.941227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:19.941369 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:19.941766 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:19.941772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:19.942018 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 04:45:19.942207 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:45:19.942726 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":103,"threads_started":1}
I20260924 04:45:19.944862 18985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 04:45:19.945132 18985 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:19.945640 18985 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 04:45:19.946779 18526 ts_manager.cc:194] Registered new tserver with Master: 86f28d9c5abe46d3964a969cb51649a3 (127.0.98.3:40643)
I20260924 04:45:19.947517 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:45:19.947551 18526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54749
I20260924 04:45:19.957125 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.949692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.98.62:39863)
0924 04:45:19.949803 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.949807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.949982 (+ 175us) server_negotiation.cc:407] Connection header received
0924 04:45:19.950073 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.950075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.950110 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.950175 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:19.950833 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.951406 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.952225 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.952420 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.952491 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.953975 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:19.953978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:19.953980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:19.954002 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:19.955528 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.955979 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.955981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.955982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.956051 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:19.956326 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:19.956328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:19.956329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:19.956516 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:19.956684 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.956900 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.956976 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:45:19.960180 18526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:45:19.963277 18526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:45:19.977927 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.971692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39577 (local address 127.0.0.1:34292)
0924 04:45:19.972261 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:19.972277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:19.972358 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:19.974138 (+ 1780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:19.974150 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:19.974165 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:19.974273 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.974283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.975328 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.975332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.976417 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.976426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.977383 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.977386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.977532 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.977567 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:45:19.977660 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:45:19.977746 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":26,"thread_start_us":120,"threads_started":1}
I20260924 04:45:19.977985 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.972770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.0.98.2:39577)
0924 04:45:19.973712 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.973724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.973747 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:45:19.973822 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.973829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.974003 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.974151 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:19.974397 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.975188 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.976539 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.977265 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.977376 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.977519 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.977717 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.977827 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":115,"threads_started":1}
I20260924 04:45:19.979173 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.972292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42767 (local address 127.0.0.1:39522)
0924 04:45:19.972546 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:19.972561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:19.972646 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:19.973665 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:19.973668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:19.973686 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:19.973799 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.973805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.975194 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.975197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.976793 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.976804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.978728 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.978732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.978833 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.978855 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:45:19.978945 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:45:19.979021 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":241,"thread_start_us":118,"threads_started":1}
I20260924 04:45:19.979434 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.972815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.0.98.1:42767)
0924 04:45:19.973191 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.973205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.973222 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:45:19.973281 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.973285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.973510 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.973677 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:19.973925 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.975021 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.978031 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.978601 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.978729 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.978879 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.978983 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.979075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":81,"threads_started":1}
I20260924 04:45:19.979483 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.971695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.0.98.3:40643)
0924 04:45:19.972908 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:19.972922 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:19.972945 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:45:19.973022 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:19.973029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:19.973243 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:19.973369 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:19.973733 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.974972 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.976922 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.977732 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.977842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.977963 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:19.979241 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0924 04:45:19.979339 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":98,"threads_started":1}
I20260924 04:45:19.979688 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:19.971590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40643 (local address 127.0.0.1:56524)
0924 04:45:19.972049 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:19.972095 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:19.972226 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:19.973393 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:19.973399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:19.973421 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:19.973587 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.973598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.975194 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.975197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.976794 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:19.976804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:19.977866 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:19.977869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:19.977967 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:19.977997 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:45:19.979014 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0924 04:45:19.979092 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":115,"thread_start_us":85,"threads_started":1}
I20260924 04:45:19.981242 18785 tablet_service.cc:1467] Processing CreateTablet for tablet aeff79901bc94481844d3e20fe2ffd6a (DEFAULT_TABLE table=test-table [id=d17f405abd1446fea4d600900e188dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:19.982237 18785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff79901bc94481844d3e20fe2ffd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:19.982190 18919 tablet_service.cc:1467] Processing CreateTablet for tablet aeff79901bc94481844d3e20fe2ffd6a (DEFAULT_TABLE table=test-table [id=d17f405abd1446fea4d600900e188dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:19.983058 18919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff79901bc94481844d3e20fe2ffd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:19.986334 18651 tablet_service.cc:1467] Processing CreateTablet for tablet aeff79901bc94481844d3e20fe2ffd6a (DEFAULT_TABLE table=test-table [id=d17f405abd1446fea4d600900e188dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:19.987419 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeff79901bc94481844d3e20fe2ffd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:19.990163 19002 tablet_bootstrap.cc:492] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Bootstrap starting.
I20260924 04:45:19.991487 19003 tablet_bootstrap.cc:492] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Bootstrap starting.
I20260924 04:45:19.992213 19002 tablet_bootstrap.cc:654] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:19.993268 19002 log.cc:824] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:19.993729 19003 tablet_bootstrap.cc:654] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:19.994961 19003 log.cc:824] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:19.995704 19002 tablet_bootstrap.cc:492] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: No bootstrap required, opened a new log
I20260924 04:45:19.995694 19005 tablet_bootstrap.cc:492] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce: Bootstrap starting.
I20260924 04:45:19.995954 19002 ts_tablet_manager.cc:1397] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:45:19.997893 19005 tablet_bootstrap.cc:654] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:19.999231 19005 log.cc:824] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:19.999854 19003 tablet_bootstrap.cc:492] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: No bootstrap required, opened a new log
I20260924 04:45:20.000079 19003 ts_tablet_manager.cc:1397] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 04:45:20.000521 19002 raft_consensus.cc:348] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.000792 19002 raft_consensus.cc:374] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:20.000864 19002 raft_consensus.cc:729] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f28d9c5abe46d3964a969cb51649a3, State: Initialized, Role: FOLLOWER
I20260924 04:45:20.001313 19005 tablet_bootstrap.cc:492] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce: No bootstrap required, opened a new log
I20260924 04:45:20.001348 19002 consensus_queue.cc:260] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.001513 19005 ts_tablet_manager.cc:1397] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:45:20.002102 18985 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260924 04:45:20.002538 19002 ts_tablet_manager.cc:1428] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:45:20.005421 19005 raft_consensus.cc:348] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.005421 19003 raft_consensus.cc:348] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.005728 19005 raft_consensus.cc:374] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:20.005759 19003 raft_consensus.cc:374] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:20.005800 19005 raft_consensus.cc:729] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce3903ac1e84d7aad47b20b8b7933ce, State: Initialized, Role: FOLLOWER
I20260924 04:45:20.005811 19003 raft_consensus.cc:729] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad38b166fbd949a3856473584cb64ee4, State: Initialized, Role: FOLLOWER
I20260924 04:45:20.006306 19005 consensus_queue.cc:260] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.006306 19003 consensus_queue.cc:260] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.007014 18717 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260924 04:45:20.007452 19005 ts_tablet_manager.cc:1428] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:45:20.007452 19003 ts_tablet_manager.cc:1428] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:45:20.007843 18851 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260924 04:45:20.014492 19010 raft_consensus.cc:482] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:20.014714 19010 raft_consensus.cc:504] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.016021 19010 leader_election.cc:288] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86f28d9c5abe46d3964a969cb51649a3 (127.0.98.3:40643), 6ce3903ac1e84d7aad47b20b8b7933ce (127.0.98.1:42767)
I20260924 04:45:20.019847 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.016349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56077 (local address 127.0.98.3:40643)
0924 04:45:20.016482 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.016487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.016501 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:45:20.016671 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.016674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.016722 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.016823 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:20.017093 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.017658 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.018780 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.019267 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.019386 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.019449 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.019637 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.019709 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:45:20.019847 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.016243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40643 (local address 127.0.98.2:56077)
0924 04:45:20.016429 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:20.016444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:20.016538 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:20.016839 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:20.016842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:20.016860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:20.016979 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.016984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.017783 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.017787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.018637 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.018653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.019392 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.019395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.019502 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.019530 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:45:20.019620 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:45:20.019709 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 04:45:20.019949 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.016401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36531 (local address 127.0.98.1:42767)
0924 04:45:20.016524 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.016528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.016794 (+ 266us) server_negotiation.cc:407] Connection header received
0924 04:45:20.016924 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.016927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.016977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.017063 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:20.017317 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.017846 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.018895 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.019381 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.019498 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.019561 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.019749 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.019823 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:20.020056 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.016292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42767 (local address 127.0.98.2:36531)
0924 04:45:20.016704 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:20.016715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:20.016825 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:20.017074 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:20.017077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:20.017092 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:20.017208 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.017213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.017962 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.017967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.018767 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.018776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.019501 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.019505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.019620 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.019647 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:45:20.019734 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:45:20.019803 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":129,"threads_started":1}
I20260924 04:45:20.020711 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff79901bc94481844d3e20fe2ffd6a" candidate_uuid: "ad38b166fbd949a3856473584cb64ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" is_pre_election: true
I20260924 04:45:20.020848 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff79901bc94481844d3e20fe2ffd6a" candidate_uuid: "ad38b166fbd949a3856473584cb64ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f28d9c5abe46d3964a969cb51649a3" is_pre_election: true
I20260924 04:45:20.021085 18671 raft_consensus.cc:2454] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad38b166fbd949a3856473584cb64ee4 in term 0.
I20260924 04:45:20.021174 18939 raft_consensus.cc:2454] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad38b166fbd949a3856473584cb64ee4 in term 0.
I20260924 04:45:20.021749 18738 leader_election.cc:302] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ce3903ac1e84d7aad47b20b8b7933ce, 86f28d9c5abe46d3964a969cb51649a3, ad38b166fbd949a3856473584cb64ee4; no voters:
I20260924 04:45:20.022086 19010 raft_consensus.cc:2781] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:45:20.022172 19010 raft_consensus.cc:482] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:45:20.022215 19010 raft_consensus.cc:3037] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:20.023435 19010 raft_consensus.cc:504] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.023854 19010 leader_election.cc:288] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [CANDIDATE]: Term 1 election: Requested vote from peers 86f28d9c5abe46d3964a969cb51649a3 (127.0.98.3:40643), 6ce3903ac1e84d7aad47b20b8b7933ce (127.0.98.1:42767)
I20260924 04:45:20.024199 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff79901bc94481844d3e20fe2ffd6a" candidate_uuid: "ad38b166fbd949a3856473584cb64ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f28d9c5abe46d3964a969cb51649a3"
I20260924 04:45:20.024248 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeff79901bc94481844d3e20fe2ffd6a" candidate_uuid: "ad38b166fbd949a3856473584cb64ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce"
I20260924 04:45:20.024312 18939 raft_consensus.cc:3037] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:20.024397 18671 raft_consensus.cc:3037] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:20.025455 18671 raft_consensus.cc:2454] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad38b166fbd949a3856473584cb64ee4 in term 1.
I20260924 04:45:20.025496 18939 raft_consensus.cc:2454] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad38b166fbd949a3856473584cb64ee4 in term 1.
I20260924 04:45:20.025807 18740 leader_election.cc:302] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce3903ac1e84d7aad47b20b8b7933ce, ad38b166fbd949a3856473584cb64ee4; no voters:
I20260924 04:45:20.026027 19010 raft_consensus.cc:2781] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:20.026273 19010 raft_consensus.cc:686] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 1 LEADER]: Becoming Leader. State: Replica: ad38b166fbd949a3856473584cb64ee4, State: Running, Role: LEADER
I20260924 04:45:20.026649 19010 consensus_queue.cc:237] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } }
I20260924 04:45:20.029978 18525 catalog_manager.cc:5495] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad38b166fbd949a3856473584cb64ee4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ad38b166fbd949a3856473584cb64ee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad38b166fbd949a3856473584cb64ee4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } health_report { overall_health: UNKNOWN } } }
I20260924 04:45:20.044343 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.040436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.0.98.2:39577)
0924 04:45:20.040598 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.040602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.040625 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:45:20.040759 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.040761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.040816 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.040913 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:20.041160 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.041733 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.042651 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.042866 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.042966 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.043379 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:20.043859 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:20.043982 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.044176 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.044214 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:45:20.050896 18671 raft_consensus.cc:1261] T aeff79901bc94481844d3e20fe2ffd6a P 6ce3903ac1e84d7aad47b20b8b7933ce [term 1 FOLLOWER]: Refusing update from remote peer ad38b166fbd949a3856473584cb64ee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:20.050897 18939 raft_consensus.cc:1261] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 1 FOLLOWER]: Refusing update from remote peer ad38b166fbd949a3856473584cb64ee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:20.051702 19012 consensus_queue.cc:1035] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86f28d9c5abe46d3964a969cb51649a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:20.051983 19010 consensus_queue.cc:1035] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce3903ac1e84d7aad47b20b8b7933ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:20.059733 19015 mvcc.cc:205] Tried to move back new op lower bound from 7332762091720261632 to 7332762091632328704. Current Snapshot: MvccSnapshot[applied={T|T < 7332762091720261632}]
I20260924 04:45:20.060259 19017 mvcc.cc:205] Tried to move back new op lower bound from 7332762091720261632 to 7332762091632328704. Current Snapshot: MvccSnapshot[applied={T|T < 7332762091720261632}]
I20260924 04:45:20.069212 18526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:20.069365 18526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:20.071904 18526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7f6820a9f7e4f039c33650fa1c660b6: Sending DeleteTablet for 3 replicas of tablet aeff79901bc94481844d3e20fe2ffd6a
I20260924 04:45:20.072773 18785 tablet_service.cc:1514] Processing DeleteTablet for tablet aeff79901bc94481844d3e20fe2ffd6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34292
I20260924 04:45:20.072927 18919 tablet_service.cc:1514] Processing DeleteTablet for tablet aeff79901bc94481844d3e20fe2ffd6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56524
I20260924 04:45:20.073374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18588
I20260924 04:45:20.073451 18651 tablet_service.cc:1514] Processing DeleteTablet for tablet aeff79901bc94481844d3e20fe2ffd6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39522
I20260924 04:45:20.074185 19026 tablet_replica.cc:330] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: stopping tablet replica
I20260924 04:45:20.074553 19026 raft_consensus.cc:2229] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:20.074882 19026 raft_consensus.cc:2258] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:20.075282 19025 tablet_replica.cc:330] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: stopping tablet replica
I20260924 04:45:20.075577 19025 raft_consensus.cc:2229] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:45:20.075941 19025 raft_consensus.cc:2258] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:20.077399 19025 ts_tablet_manager.cc:1905] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:20.078863 19026 ts_tablet_manager.cc:1905] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:20.080236 19025 ts_tablet_manager.cc:1918] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:20.080406 19025 log.cc:1196] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/aeff79901bc94481844d3e20fe2ffd6a
I20260924 04:45:20.080816 19025 ts_tablet_manager.cc:1939] T aeff79901bc94481844d3e20fe2ffd6a P ad38b166fbd949a3856473584cb64ee4: Deleting consensus metadata
I20260924 04:45:20.082126 18512 catalog_manager.cc:4841] TS ad38b166fbd949a3856473584cb64ee4 (127.0.98.2:39577): tablet aeff79901bc94481844d3e20fe2ffd6a (table test-table [id=d17f405abd1446fea4d600900e188dc6]) successfully deleted
I20260924 04:45:20.082401 19026 ts_tablet_manager.cc:1918] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:20.082506 19026 log.cc:1196] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/aeff79901bc94481844d3e20fe2ffd6a
I20260924 04:45:20.082846 19026 ts_tablet_manager.cc:1939] T aeff79901bc94481844d3e20fe2ffd6a P 86f28d9c5abe46d3964a969cb51649a3: Deleting consensus metadata
W20260924 04:45:20.084329 18512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42767 (error 108)
I20260924 04:45:20.084355 18510 catalog_manager.cc:4841] TS 86f28d9c5abe46d3964a969cb51649a3 (127.0.98.3:40643): tablet aeff79901bc94481844d3e20fe2ffd6a (table test-table [id=d17f405abd1446fea4d600900e188dc6]) successfully deleted
I20260924 04:45:20.084774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18722
I20260924 04:45:20.086460 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.086122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42767 (local address 127.0.0.1:39536)
0924 04:45:20.086259 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:20.086344 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 04:45:20.086669 18512 catalog_manager.cc:4584] TS 6ce3903ac1e84d7aad47b20b8b7933ce (127.0.98.1:42767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aeff79901bc94481844d3e20fe2ffd6a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42767: connect: Connection refused (error 111)
I20260924 04:45:20.091053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18856
I20260924 04:45:20.096841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 18494
2026-09-24T04:45:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2286 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T04:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:20Z Disabled control of system clock
I20260924 04:45:20.127907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45001
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:45:20.239811 19031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:20.240109 19031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:20.240145 19031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:20.244383 19031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:20.244451 19031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:20.244467 19031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:20.244484 19031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:20.244500 19031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:20.249511 19031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:20.250594 19031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:20.251958 19031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:45:20.260937 19031 server_base.cc:1034] running on GCE node
W20260924 04:45:20.260880 19036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:20.260918 19037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:20.260880 19039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:20.261814 19031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:20.262967 19031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:20.264173 19031 hybrid_clock.cc:648] HybridClock initialized: now 1790225120264153 us; error 35 us; skew 500 ppm
I20260924 04:45:20.266196 19031 webserver.cc:460] Webserver started at http://127.0.98.62:37491/ using document root <none> and password file <none>
I20260924 04:45:20.266821 19031 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:20.266884 19031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:20.267161 19031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:20.269337 19031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "a7c500308de04b928dc4896475dba56b"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "d3c7ad39243495ee20165af2dcae21a3"
server_key_iv: "fad375f8e89d40596d101f2afbf28aea"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.269942 19031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7c500308de04b928dc4896475dba56b"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "d3c7ad39243495ee20165af2dcae21a3"
server_key_iv: "fad375f8e89d40596d101f2afbf28aea"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.274134 19031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:20.276940 19045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.278363 19031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:20.278529 19031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "a7c500308de04b928dc4896475dba56b"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "d3c7ad39243495ee20165af2dcae21a3"
server_key_iv: "fad375f8e89d40596d101f2afbf28aea"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.278656 19031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:20.309898 19031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:20.310695 19031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:20.310909 19031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:20.318306 19031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45001
I20260924 04:45:20.318343 19097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45001 every 8 connection(s)
I20260924 04:45:20.319469 19031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:20.322839 19098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:20.324342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19031
I20260924 04:45:20.324504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:20.324795 392 external_mini_cluster.cc:1428] Setting key f9ed87130e1ebfc40a3c70d8f6840b89
I20260924 04:45:20.329520 19098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b: Bootstrap starting.
I20260924 04:45:20.332265 19098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:20.333138 19098 log.cc:824] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:20.334772 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.326331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.0.98.62:45001)
0924 04:45:20.326964 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.326980 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.327016 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:45:20.327758 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.327775 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.328165 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.328452 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:20.328847 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.329874 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.331116 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.331714 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.331835 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.332812 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:20.332817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:20.332830 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:20.333247 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:20.333357 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:20.333380 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:20.333584 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.333747 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.334050 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260924 04:45:20.335395 19098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b: No bootstrap required, opened a new log
I20260924 04:45:20.339776 19098 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c500308de04b928dc4896475dba56b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260924 04:45:20.340060 19098 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:20.340121 19098 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c500308de04b928dc4896475dba56b, State: Initialized, Role: FOLLOWER
I20260924 04:45:20.340536 19098 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c500308de04b928dc4896475dba56b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260924 04:45:20.340684 19098 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:20.340771 19098 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:20.340906 19098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:20.341884 19098 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c500308de04b928dc4896475dba56b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260924 04:45:20.342231 19098 leader_election.cc:302] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7c500308de04b928dc4896475dba56b; no voters:
I20260924 04:45:20.342583 19098 leader_election.cc:288] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:20.342749 19103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:20.342999 19103 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [term 1 LEADER]: Becoming Leader. State: Replica: a7c500308de04b928dc4896475dba56b, State: Running, Role: LEADER
I20260924 04:45:20.343364 19103 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c500308de04b928dc4896475dba56b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260924 04:45:20.343972 19098 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:20.345188 19104 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7c500308de04b928dc4896475dba56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c500308de04b928dc4896475dba56b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } }
I20260924 04:45:20.345320 19104 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:20.345590 19105 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c500308de04b928dc4896475dba56b. Latest consensus state: current_term: 1 leader_uuid: "a7c500308de04b928dc4896475dba56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c500308de04b928dc4896475dba56b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } }
I20260924 04:45:20.345734 19112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:20.345768 19105 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:20.349015 19112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:20.355146 19112 catalog_manager.cc:1347] Generated new cluster ID: 738d24b29a4a446789659725ca91bccb
I20260924 04:45:20.355223 19112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:20.369930 19112 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:20.370893 19112 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:20.379702 19112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b: Generated new TSK 0
I20260924 04:45:20.380525 19112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:20.388792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:45:20.506307 19122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:20.506620 19122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:20.506701 19122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:20.510862 19122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:20.510975 19122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:20.511073 19122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:45:20.516310 19122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:20.517558 19122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:20.519014 19122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:20.529277 19127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:20.529376 19122 server_base.cc:1034] running on GCE node
W20260924 04:45:20.529299 19130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:20.529312 19128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:20.530005 19122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:20.530696 19122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:20.531886 19122 hybrid_clock.cc:648] HybridClock initialized: now 1790225120531814 us; error 86 us; skew 500 ppm
I20260924 04:45:20.534053 19122 webserver.cc:460] Webserver started at http://127.0.98.1:39757/ using document root <none> and password file <none>
I20260924 04:45:20.534670 19122 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:20.534736 19122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:20.534960 19122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:20.537174 19122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "d602eac17d9c4e2a9617a66908fcbee0"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "4b23a7ebfbc17569bfbd9b57f527cdf0"
server_key_iv: "597a9ac1faabc0c4b88eacea28917532"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.537791 19122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d602eac17d9c4e2a9617a66908fcbee0"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "4b23a7ebfbc17569bfbd9b57f527cdf0"
server_key_iv: "597a9ac1faabc0c4b88eacea28917532"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.541949 19122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:20.544837 19136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.546458 19122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:20.546643 19122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "d602eac17d9c4e2a9617a66908fcbee0"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "4b23a7ebfbc17569bfbd9b57f527cdf0"
server_key_iv: "597a9ac1faabc0c4b88eacea28917532"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.546782 19122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:20.562791 19122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:20.563623 19122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:20.563843 19122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:20.564476 19122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:20.565479 19122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:20.565554 19122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.565634 19122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:20.565678 19122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.575253 19122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43371
I20260924 04:45:20.575270 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43371 every 8 connection(s)
I20260924 04:45:20.576197 19122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:45:20.585938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19122
I20260924 04:45:20.585981 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.578185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46707 (local address 127.0.98.62:45001)
0924 04:45:20.578336 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.578340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.579446 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 04:45:20.580904 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.580908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.580973 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.581056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:20.581969 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.582652 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.583909 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.584153 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.584273 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.585429 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:20.585433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:20.585435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:20.585466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:20.585491 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:20.585498 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:20.585575 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.585761 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.585828 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:45:20.586112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:45:20.586448 392 external_mini_cluster.cc:1428] Setting key 61098dc1d1eb5f439597b17ddf0de7da
I20260924 04:45:20.586884 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.578593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:46707)
0924 04:45:20.579295 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:20.579336 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:20.580690 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:20.581223 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:20.581230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:20.581313 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:20.581818 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.581830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.582786 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.582790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.583779 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.583789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.584314 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.584318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.584483 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.585151 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:20.585152 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:20.585192 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:20.585256 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:20.585589 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:20.585608 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:20.585619 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 04:45:20.585747 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 04:45:20.586036 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260924 04:45:20.588596 19250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260924 04:45:20.588872 19250 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:20.589051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:45:20.589550 19250 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260924 04:45:20.591346 19061 ts_manager.cc:194] Registered new tserver with Master: d602eac17d9c4e2a9617a66908fcbee0 (127.0.98.1:43371)
I20260924 04:45:20.592702 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46707
W20260924 04:45:20.704749 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:20.705060 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:20.705147 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:20.709188 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:20.709280 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:20.709371 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:45:20.714273 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:20.715425 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:20.716807 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:20.726508 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:20.726624 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:20.726620 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:20.726887 19253 server_base.cc:1034] running on GCE node
I20260924 04:45:20.727336 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:20.727914 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:20.729108 19253 hybrid_clock.cc:648] HybridClock initialized: now 1790225120729093 us; error 28 us; skew 500 ppm
I20260924 04:45:20.731106 19253 webserver.cc:460] Webserver started at http://127.0.98.2:38671/ using document root <none> and password file <none>
I20260924 04:45:20.731817 19253 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:20.731894 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:20.732131 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:20.734269 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "196f3b670da34710a7034c52f3ec5209"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "52d1a6164bfe84bd90023485546d6aaf"
server_key_iv: "57442358f8146f6b3502b57b54475692"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.734884 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "196f3b670da34710a7034c52f3ec5209"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "52d1a6164bfe84bd90023485546d6aaf"
server_key_iv: "57442358f8146f6b3502b57b54475692"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.738732 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:45:20.741466 19267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.742832 19253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:20.742973 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "196f3b670da34710a7034c52f3ec5209"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "52d1a6164bfe84bd90023485546d6aaf"
server_key_iv: "57442358f8146f6b3502b57b54475692"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.743084 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:20.769618 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:20.770359 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:20.770653 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:20.771327 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:20.772348 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:20.772393 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.772433 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:20.772459 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.781909 19253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41427
I20260924 04:45:20.781914 19380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41427 every 8 connection(s)
I20260924 04:45:20.782866 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:45:20.785969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19253
I20260924 04:45:20.786152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:45:20.786487 392 external_mini_cluster.cc:1428] Setting key 78fb8c3c61d4ae97ba281eaf7e474085
I20260924 04:45:20.788928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:45:20.793228 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.784943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54313 (local address 127.0.98.62:45001)
0924 04:45:20.785084 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.785088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.786313 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 04:45:20.788170 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.788173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.788223 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.788306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:20.789311 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.789841 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.791101 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.791379 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.791499 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.792700 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:20.792704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:20.792706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:20.792721 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:20.792748 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:20.792756 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:20.792833 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.793004 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.793070 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:45:20.794168 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.785326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:54313)
0924 04:45:20.786137 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:20.786195 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:20.787945 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:20.788509 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:20.788516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:20.788617 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:20.789155 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.789175 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.789973 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.789976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.790950 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.790959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.791543 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.791547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.791737 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.792399 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:20.792400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:20.792445 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:20.792514 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:20.792849 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:20.792858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:20.792864 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:45:20.793005 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:45:20.793313 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
I20260924 04:45:20.795435 19381 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260924 04:45:20.795732 19381 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:20.796449 19381 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260924 04:45:20.797775 19061 ts_manager.cc:194] Registered new tserver with Master: 196f3b670da34710a7034c52f3ec5209 (127.0.98.2:41427)
I20260924 04:45:20.798671 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54313
W20260924 04:45:20.904013 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:20.904349 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:20.904392 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:20.908398 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:20.908455 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:20.908512 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:45:20.913203 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:20.914230 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:20.915702 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:20.925246 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:20.925204 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:20.925279 19384 server_base.cc:1034] running on GCE node
W20260924 04:45:20.925287 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:20.926066 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:20.926774 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:20.927958 19384 hybrid_clock.cc:648] HybridClock initialized: now 1790225120927933 us; error 39 us; skew 500 ppm
I20260924 04:45:20.929876 19384 webserver.cc:460] Webserver started at http://127.0.98.3:37577/ using document root <none> and password file <none>
I20260924 04:45:20.930467 19384 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:20.930562 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:20.930796 19384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:20.932962 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d7ad40231114dae8ac141b5849d9101"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "6a245912c9c9d223f7d0081ff74419d5"
server_key_iv: "53a34a1dcd7f1e4c6ae64c3b6c6a3888"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.933552 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d7ad40231114dae8ac141b5849d9101"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "6a245912c9c9d223f7d0081ff74419d5"
server_key_iv: "53a34a1dcd7f1e4c6ae64c3b6c6a3888"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.937490 19384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:20.940019 19398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.941370 19384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:45:20.941514 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "0d7ad40231114dae8ac141b5849d9101"
format_stamp: "Formatted at 2026-09-24 04:45:20 on dist-test-slave-npg2"
server_key: "6a245912c9c9d223f7d0081ff74419d5"
server_key_iv: "53a34a1dcd7f1e4c6ae64c3b6c6a3888"
server_key_version: "encryptionkey@0"
I20260924 04:45:20.941638 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:20.963443 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:20.964177 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:20.964381 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:20.964972 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:20.965939 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:20.966012 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.966084 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:20.966142 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:20.975605 19384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38987
I20260924 04:45:20.975625 19511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38987 every 8 connection(s)
I20260924 04:45:20.976543 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:45:20.985673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19384
I20260924 04:45:20.985781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:45:20.986049 392 external_mini_cluster.cc:1428] Setting key 400e7338e3e3f809ddfa2235dd6e33ff
I20260924 04:45:20.986130 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.978380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48363 (local address 127.0.98.62:45001)
0924 04:45:20.978536 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:20.978540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:20.979682 (+ 1142us) server_negotiation.cc:407] Connection header received
0924 04:45:20.981057 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:20.981059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:20.981104 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:20.981186 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:20.982185 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.982810 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.984067 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.984304 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.984383 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.985516 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:20.985521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:20.985526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:20.985542 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:20.985565 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:20.985575 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:20.985656 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:20.985894 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 04:45:20.985956 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:45:20.986994 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:20.978819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:48363)
0924 04:45:20.979542 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:20.979583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:20.980851 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:20.981342 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:20.981349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:20.981434 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:20.982022 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.982039 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.982924 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.982927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.983957 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:20.983967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:20.984437 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:20.984440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:20.984586 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:20.985248 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:20.985249 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:20.985289 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:20.985369 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:20.985699 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:20.985706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:20.985712 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:45:20.985887 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 04:45:20.986164 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260924 04:45:20.988086 19512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260924 04:45:20.988338 19512 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:20.988862 19512 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260924 04:45:20.989933 19061 ts_manager.cc:194] Registered new tserver with Master: 0d7ad40231114dae8ac141b5849d9101 (127.0.98.3:38987)
I20260924 04:45:20.990521 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48363
I20260924 04:45:21.000383 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:45:21.006995 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.002778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.0.98.62:45001)
0924 04:45:21.002989 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.002995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.003179 (+ 184us) server_negotiation.cc:407] Connection header received
0924 04:45:21.003353 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.003356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.003406 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.003580 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:21.003871 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.004391 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.005244 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.005464 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.005588 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.006485 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:21.006489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:21.006496 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:21.006509 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:21.006527 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:21.006535 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:21.006668 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.006794 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.006836 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 04:45:21.010111 19061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:45:21.013388 19061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:45:21.029780 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.023859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38987 (local address 127.0.0.1:54088)
0924 04:45:21.024969 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.025021 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:21.025175 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:21.025828 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:21.025839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:21.025865 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:21.026021 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.026031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.027536 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.027542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.028436 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.028445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.029229 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.029233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.029336 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.029370 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 04:45:21.029498 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 04:45:21.029616 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":73,"threads_started":1}
I20260924 04:45:21.029786 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.023997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.0.98.3:38987)
0924 04:45:21.024358 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.024370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.025329 (+ 959us) server_negotiation.cc:407] Connection header received
0924 04:45:21.025412 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.025417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.025671 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.025825 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:21.026164 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.027405 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.028570 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.029104 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.029230 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.029422 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.029524 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.029620 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20260924 04:45:21.030793 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.024589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41427 (local address 127.0.0.1:42218)
0924 04:45:21.025236 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.025252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:21.025345 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:21.026070 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:21.026076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:21.026097 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:21.026200 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.026209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.027594 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.027597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.029110 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.029120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.030149 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.030153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.030475 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.030502 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:45:21.030580 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:45:21.030663 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":152,"thread_start_us":100,"threads_started":1}
I20260924 04:45:21.030889 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.024773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.0.98.2:41427)
0924 04:45:21.025199 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.025211 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.025444 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:45:21.025530 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.025536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.025780 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.025922 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:21.026326 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.027491 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.029252 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.030027 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.030146 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.030328 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.030612 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.030710 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":96,"threads_started":1}
I20260924 04:45:21.031499 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.024946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43371 (local address 127.0.0.1:40276)
0924 04:45:21.025515 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.025528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:21.025612 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:21.026029 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:21.026032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:21.026049 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:21.026179 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.026186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.027477 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.027482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.029110 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.029120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.031057 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.031061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.031193 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.031221 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:45:21.031294 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:45:21.031366 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":248,"thread_start_us":97,"threads_started":1}
I20260924 04:45:21.031958 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.025107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.98.1:43371)
0924 04:45:21.025455 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.025466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.025615 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:45:21.025712 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.025719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.025883 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.026034 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:21.026295 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.027367 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.029252 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.030704 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.030811 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.030987 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.031689 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.031786 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":151,"threads_started":1}
I20260924 04:45:21.033401 19446 tablet_service.cc:1467] Processing CreateTablet for tablet 3c568e45d8a04629a07956ec6f6235b4 (DEFAULT_TABLE table=test-table [id=58b89705fe194bc0874316c832ace04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:21.033813 19315 tablet_service.cc:1467] Processing CreateTablet for tablet 3c568e45d8a04629a07956ec6f6235b4 (DEFAULT_TABLE table=test-table [id=58b89705fe194bc0874316c832ace04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:21.034333 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c568e45d8a04629a07956ec6f6235b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:21.034804 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c568e45d8a04629a07956ec6f6235b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:21.034494 19184 tablet_service.cc:1467] Processing CreateTablet for tablet 3c568e45d8a04629a07956ec6f6235b4 (DEFAULT_TABLE table=test-table [id=58b89705fe194bc0874316c832ace04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:21.035447 19184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c568e45d8a04629a07956ec6f6235b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:21.040351 19529 tablet_bootstrap.cc:492] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Bootstrap starting.
I20260924 04:45:21.042610 19530 tablet_bootstrap.cc:492] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: Bootstrap starting.
I20260924 04:45:21.042768 19529 tablet_bootstrap.cc:654] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:21.043555 19531 tablet_bootstrap.cc:492] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Bootstrap starting.
I20260924 04:45:21.044063 19529 log.cc:824] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:21.045091 19530 tablet_bootstrap.cc:654] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:21.045972 19530 log.cc:824] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:21.046114 19531 tablet_bootstrap.cc:654] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:21.046576 19529 tablet_bootstrap.cc:492] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: No bootstrap required, opened a new log
I20260924 04:45:21.046778 19529 ts_tablet_manager.cc:1397] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:45:21.047042 19531 log.cc:824] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:21.048870 19531 tablet_bootstrap.cc:492] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: No bootstrap required, opened a new log
I20260924 04:45:21.049093 19531 ts_tablet_manager.cc:1397] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 04:45:21.049129 19530 tablet_bootstrap.cc:492] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: No bootstrap required, opened a new log
I20260924 04:45:21.049304 19530 ts_tablet_manager.cc:1397] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:45:21.051393 19529 raft_consensus.cc:348] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.051682 19529 raft_consensus.cc:374] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:21.051775 19529 raft_consensus.cc:729] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7ad40231114dae8ac141b5849d9101, State: Initialized, Role: FOLLOWER
I20260924 04:45:21.052258 19529 consensus_queue.cc:260] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.052934 19512 heartbeater.cc:503] Master 127.0.98.62:45001 was elected leader, sending a full tablet report...
I20260924 04:45:21.052986 19531 raft_consensus.cc:348] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.053217 19531 raft_consensus.cc:374] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:21.053272 19531 raft_consensus.cc:729] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196f3b670da34710a7034c52f3ec5209, State: Initialized, Role: FOLLOWER
I20260924 04:45:21.053304 19529 ts_tablet_manager.cc:1428] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260924 04:45:21.053539 19530 raft_consensus.cc:348] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.053778 19530 raft_consensus.cc:374] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:21.053737 19531 consensus_queue.cc:260] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.053853 19530 raft_consensus.cc:729] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d602eac17d9c4e2a9617a66908fcbee0, State: Initialized, Role: FOLLOWER
I20260924 04:45:21.054266 19530 consensus_queue.cc:260] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.054493 19381 heartbeater.cc:503] Master 127.0.98.62:45001 was elected leader, sending a full tablet report...
I20260924 04:45:21.055209 19530 ts_tablet_manager.cc:1428] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:45:21.055474 19531 ts_tablet_manager.cc:1428] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 04:45:21.055503 19250 heartbeater.cc:503] Master 127.0.98.62:45001 was elected leader, sending a full tablet report...
W20260924 04:45:21.077459 19251 tablet.cc:2369] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:45:21.227510 19513 tablet.cc:2369] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:21.242406 19536 raft_consensus.cc:482] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:21.242666 19536 raft_consensus.cc:504] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.243949 19536 leader_election.cc:288] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7ad40231114dae8ac141b5849d9101 (127.0.98.3:38987), d602eac17d9c4e2a9617a66908fcbee0 (127.0.98.1:43371)
I20260924 04:45:21.248543 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.244417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49959 (local address 127.0.98.3:38987)
0924 04:45:21.244632 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.244637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.244653 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:45:21.244709 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.244712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.244796 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.244877 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:21.245346 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.246114 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.247355 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.247913 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.248007 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.248100 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.248331 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.248410 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260924 04:45:21.248548 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.244404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51847 (local address 127.0.98.1:43371)
0924 04:45:21.244540 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.244547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.244806 (+ 259us) server_negotiation.cc:407] Connection header received
0924 04:45:21.244961 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.244965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.245020 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.245099 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:21.245629 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.246219 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.247572 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.247961 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.248093 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.248152 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.248374 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.248445 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:45:21.248543 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.244182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38987 (local address 127.0.98.2:49959)
0924 04:45:21.244339 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.244356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:21.244455 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:21.245013 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:21.245016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:21.245031 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:21.245194 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.245202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.246219 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.246223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.247241 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.247250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.248044 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.248051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.248198 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.248220 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:45:21.248289 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:45:21.248386 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":1}
I20260924 04:45:21.248569 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.244282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43371 (local address 127.0.98.2:51847)
0924 04:45:21.244666 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.244678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:21.244841 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:21.245221 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:21.245223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:21.245257 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:21.245437 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.245445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.246321 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.246324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.247396 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:21.247407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.248132 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.248136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:21.248219 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.248246 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:45:21.248345 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:45:21.248411 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260924 04:45:21.249207 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c568e45d8a04629a07956ec6f6235b4" candidate_uuid: "196f3b670da34710a7034c52f3ec5209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d602eac17d9c4e2a9617a66908fcbee0" is_pre_election: true
I20260924 04:45:21.249483 19466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c568e45d8a04629a07956ec6f6235b4" candidate_uuid: "196f3b670da34710a7034c52f3ec5209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7ad40231114dae8ac141b5849d9101" is_pre_election: true
I20260924 04:45:21.249543 19204 raft_consensus.cc:2454] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196f3b670da34710a7034c52f3ec5209 in term 0.
I20260924 04:45:21.249771 19466 raft_consensus.cc:2454] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196f3b670da34710a7034c52f3ec5209 in term 0.
I20260924 04:45:21.250072 19268 leader_election.cc:302] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196f3b670da34710a7034c52f3ec5209, d602eac17d9c4e2a9617a66908fcbee0; no voters:
I20260924 04:45:21.250383 19536 raft_consensus.cc:2781] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:45:21.250458 19536 raft_consensus.cc:482] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:45:21.250510 19536 raft_consensus.cc:3037] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:21.251755 19536 raft_consensus.cc:504] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.252156 19536 leader_election.cc:288] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7ad40231114dae8ac141b5849d9101 (127.0.98.3:38987), d602eac17d9c4e2a9617a66908fcbee0 (127.0.98.1:43371)
I20260924 04:45:21.252470 19466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c568e45d8a04629a07956ec6f6235b4" candidate_uuid: "196f3b670da34710a7034c52f3ec5209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7ad40231114dae8ac141b5849d9101"
I20260924 04:45:21.252619 19466 raft_consensus.cc:3037] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:21.252646 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c568e45d8a04629a07956ec6f6235b4" candidate_uuid: "196f3b670da34710a7034c52f3ec5209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d602eac17d9c4e2a9617a66908fcbee0"
I20260924 04:45:21.252799 19204 raft_consensus.cc:3037] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:21.253690 19466 raft_consensus.cc:2454] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196f3b670da34710a7034c52f3ec5209 in term 1.
I20260924 04:45:21.253942 19204 raft_consensus.cc:2454] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196f3b670da34710a7034c52f3ec5209 in term 1.
I20260924 04:45:21.253991 19269 leader_election.cc:302] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7ad40231114dae8ac141b5849d9101, 196f3b670da34710a7034c52f3ec5209; no voters:
I20260924 04:45:21.254179 19536 raft_consensus.cc:2781] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:21.254405 19536 raft_consensus.cc:686] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 1 LEADER]: Becoming Leader. State: Replica: 196f3b670da34710a7034c52f3ec5209, State: Running, Role: LEADER
I20260924 04:45:21.254762 19536 consensus_queue.cc:237] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } }
I20260924 04:45:21.258239 19061 catalog_manager.cc:5495] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 196f3b670da34710a7034c52f3ec5209 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "196f3b670da34710a7034c52f3ec5209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "196f3b670da34710a7034c52f3ec5209" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } health_report { overall_health: HEALTHY } } }
W20260924 04:45:21.285632 19382 tablet.cc:2369] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:21.326612 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.322577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.98.2:41427)
0924 04:45:21.322720 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:21.322725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:21.322742 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:45:21.322834 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:21.322836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:21.322891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:21.323003 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:21.323348 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.324095 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.324924 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:21.325269 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:21.325362 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:21.325625 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:21.326121 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:21.326258 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:21.326412 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:45:21.326451 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:45:21.333060 19204 raft_consensus.cc:1261] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 1 FOLLOWER]: Refusing update from remote peer 196f3b670da34710a7034c52f3ec5209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:21.333087 19466 raft_consensus.cc:1261] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 1 FOLLOWER]: Refusing update from remote peer 196f3b670da34710a7034c52f3ec5209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:21.333827 19539 consensus_queue.cc:1035] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d602eac17d9c4e2a9617a66908fcbee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:21.334017 19536 consensus_queue.cc:1035] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7ad40231114dae8ac141b5849d9101" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:21.345293 19544 mvcc.cc:205] Tried to move back new op lower bound from 7332762096972591104 to 7332762096662757376. Current Snapshot: MvccSnapshot[applied={T|T < 7332762096972591104 or (T in {7332762096972591104})}]
I20260924 04:45:21.345898 19543 mvcc.cc:205] Tried to move back new op lower bound from 7332762096972591104 to 7332762096662757376. Current Snapshot: MvccSnapshot[applied={T|T < 7332762096972591104}]
I20260924 04:45:21.350407 19061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:21.350615 19061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:21.353144 19061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7c500308de04b928dc4896475dba56b: Sending DeleteTablet for 3 replicas of tablet 3c568e45d8a04629a07956ec6f6235b4
I20260924 04:45:21.354009 19184 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c568e45d8a04629a07956ec6f6235b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:21 UTC) from {username='slave'} at 127.0.0.1:40276
I20260924 04:45:21.354118 19446 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c568e45d8a04629a07956ec6f6235b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:21 UTC) from {username='slave'} at 127.0.0.1:54088
I20260924 04:45:21.354339 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c568e45d8a04629a07956ec6f6235b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:21 UTC) from {username='slave'} at 127.0.0.1:42218
I20260924 04:45:21.354571 19552 tablet_replica.cc:330] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0: stopping tablet replica
I20260924 04:45:21.354947 19552 raft_consensus.cc:2229] T 3c568e45d8a04629a07956ec6f6235b4 P d602eac17d9c4e2a9617a66908fcbee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:21.355098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19122
I20260924 04:45:21.355784 19551 tablet_replica.cc:330] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: stopping tablet replica
I20260924 04:45:21.356141 19551 raft_consensus.cc:2229] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:21.356511 19551 raft_consensus.cc:2258] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:21.358085 19551 ts_tablet_manager.cc:1905] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:21.361622 19551 ts_tablet_manager.cc:1918] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:21.361740 19551 log.cc:1196] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/3c568e45d8a04629a07956ec6f6235b4
I20260924 04:45:21.362020 19553 tablet_replica.cc:330] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: stopping tablet replica
I20260924 04:45:21.362066 19551 ts_tablet_manager.cc:1939] T 3c568e45d8a04629a07956ec6f6235b4 P 0d7ad40231114dae8ac141b5849d9101: Deleting consensus metadata
I20260924 04:45:21.362413 19553 raft_consensus.cc:2229] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:45:21.362838 19553 raft_consensus.cc:2258] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:21.363623 19047 catalog_manager.cc:4841] TS 0d7ad40231114dae8ac141b5849d9101 (127.0.98.3:38987): tablet 3c568e45d8a04629a07956ec6f6235b4 (table test-table [id=58b89705fe194bc0874316c832ace04a]) successfully deleted
I20260924 04:45:21.364567 19553 ts_tablet_manager.cc:1905] T 3c568e45d8a04629a07956ec6f6235b4 P 196f3b670da34710a7034c52f3ec5209: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:45:21.364965 19046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43371 (error 108)
I20260924 04:45:21.365825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19253
I20260924 04:45:21.366039 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.365730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43371 (local address 127.0.0.1:36090)
0924 04:45:21.365872 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.365953 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:45:21.366201 19046 catalog_manager.cc:4584] TS d602eac17d9c4e2a9617a66908fcbee0 (127.0.98.1:43371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c568e45d8a04629a07956ec6f6235b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43371: connect: Connection refused (error 111)
I20260924 04:45:21.372071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19384
I20260924 04:45:21.373337 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:21.372216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41427 (local address 127.0.0.1:33336)
0924 04:45:21.373165 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:21.373231 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":843}
I20260924 04:45:21.377617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19031
2026-09-24T04:45:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1279 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:21 dist-test-slave-npg2 krb5kdc[19557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:21 dist-test-slave-npg2 krb5kdc[19557](info): set up 2 sockets
Sep 24 04:45:21 dist-test-slave-npg2 krb5kdc[19557](info): commencing operation
krb5kdc: starting...
W20260924 04:45:22.455319 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:22 dist-test-slave-npg2 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:22.586961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:22.702234 19570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:22.702492 19570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:22.702533 19570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:22.706676 19570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:22.706744 19570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:22.706764 19570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:22.706781 19570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:22.706796 19570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:22.712317 19570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:22.713397 19570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:22.714447 19570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:22.723202 19576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:22.723278 19575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:22.723299 19570 server_base.cc:1034] running on GCE node
W20260924 04:45:22.723210 19578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:22.724115 19570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:22.725250 19570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:22.726421 19570 hybrid_clock.cc:648] HybridClock initialized: now 1790225122726399 us; error 37 us; skew 500 ppm
Sep 24 04:45:22 dist-test-slave-npg2 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:22.735262 19570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:22.736277 19570 webserver.cc:460] Webserver started at http://127.0.98.62:41961/ using document root <none> and password file <none>
I20260924 04:45:22.736912 19570 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:22.737010 19570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:22.737275 19570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:22.739464 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "ae82326f603c484cb01b5fd532768647"
format_stamp: "Formatted at 2026-09-24 04:45:22 on dist-test-slave-npg2"
server_key: "c6fb6593b606088411bbb6a77426aab7"
server_key_iv: "bf8b807abc753c0d4dc28a0522ee8382"
server_key_version: "encryptionkey@0"
I20260924 04:45:22.740041 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae82326f603c484cb01b5fd532768647"
format_stamp: "Formatted at 2026-09-24 04:45:22 on dist-test-slave-npg2"
server_key: "c6fb6593b606088411bbb6a77426aab7"
server_key_iv: "bf8b807abc753c0d4dc28a0522ee8382"
server_key_version: "encryptionkey@0"
I20260924 04:45:22.743928 19570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:45:22.746364 19585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:22.747761 19570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:22.747908 19570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "ae82326f603c484cb01b5fd532768647"
format_stamp: "Formatted at 2026-09-24 04:45:22 on dist-test-slave-npg2"
server_key: "c6fb6593b606088411bbb6a77426aab7"
server_key_iv: "bf8b807abc753c0d4dc28a0522ee8382"
server_key_version: "encryptionkey@0"
I20260924 04:45:22.748031 19570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:22.774194 19570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:22.777534 19570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:22.777801 19570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:22.785316 19570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38051
I20260924 04:45:22.785341 19637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38051 every 8 connection(s)
I20260924 04:45:22.786289 19570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:22.789143 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:22.793846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19570
I20260924 04:45:22.794034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:22.794456 392 external_mini_cluster.cc:1428] Setting key ecd14fb99c2c22ae3b919c8d5e0c809d
I20260924 04:45:22.795276 19638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647: Bootstrap starting.
I20260924 04:45:22.797785 19638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:22.798810 19638 log.cc:824] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:22.801234 19638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647: No bootstrap required, opened a new log
Sep 24 04:45:22 dist-test-slave-npg2 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:22.806118 19638 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae82326f603c484cb01b5fd532768647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38051 } }
I20260924 04:45:22.806432 19638 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:22.806484 19638 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae82326f603c484cb01b5fd532768647, State: Initialized, Role: FOLLOWER
I20260924 04:45:22.807003 19638 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae82326f603c484cb01b5fd532768647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38051 } }
I20260924 04:45:22.807188 19638 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:22.807258 19638 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:22.807361 19638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:22.808540 19638 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae82326f603c484cb01b5fd532768647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38051 } }
I20260924 04:45:22.808943 19638 leader_election.cc:302] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae82326f603c484cb01b5fd532768647; no voters:
I20260924 04:45:22.809314 19638 leader_election.cc:288] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:22.809428 19643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:22.809739 19643 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [term 1 LEADER]: Becoming Leader. State: Replica: ae82326f603c484cb01b5fd532768647, State: Running, Role: LEADER
I20260924 04:45:22.810074 19643 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae82326f603c484cb01b5fd532768647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38051 } }
I20260924 04:45:22.810181 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:22.795987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.98.62:38051)
0924 04:45:22.796435 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:22.796450 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:22.796494 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:45:22.797153 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:22.797172 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:22.797528 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:22.797831 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:22.798781 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:22.800130 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:22.801259 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:22.801787 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:22.801908 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:22.804734 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:22.804741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:22.804759 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:22.804793 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:22.806800 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:22.807321 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:22.807327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:22.807333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:22.807479 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:22.807821 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:22.807824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:22.807826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:22.808364 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:22.808575 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:22.808873 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:45:22.809211 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":150,"threads_started":1}
I20260924 04:45:22.810596 19638 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:22.811280 19643 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae82326f603c484cb01b5fd532768647. Latest consensus state: current_term: 1 leader_uuid: "ae82326f603c484cb01b5fd532768647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae82326f603c484cb01b5fd532768647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38051 } } }
I20260924 04:45:22.811407 19643 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:22.811703 19645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae82326f603c484cb01b5fd532768647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae82326f603c484cb01b5fd532768647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38051 } } }
I20260924 04:45:22.811784 19645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:22.812480 19651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:22.815377 19651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:22.821763 19651 catalog_manager.cc:1347] Generated new cluster ID: 78d7d0a5023c4dc5a13d0e91c2a445c4
I20260924 04:45:22.821839 19651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:22.840940 19651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:22.842262 19651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:22.849258 19651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647: Generated new TSK 0
I20260924 04:45:22.850224 19651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:45:22.883625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38051
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:22.997702 19664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:22.997977 19664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:22.998023 19664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:23.002205 19664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:23.002290 19664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:23.002354 19664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:45:23.008033 19664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:23.009265 19664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:23.010407 19664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:23.021062 19669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:23.021070 19670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:23.021062 19672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:23.021323 19664 server_base.cc:1034] running on GCE node
I20260924 04:45:23.021806 19664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:23.022497 19664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:23.023687 19664 hybrid_clock.cc:648] HybridClock initialized: now 1790225123023669 us; error 31 us; skew 500 ppm
Sep 24 04:45:23 dist-test-slave-npg2 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:23.026552 19664 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:45:23.027575 19664 webserver.cc:460] Webserver started at http://127.0.98.1:42507/ using document root <none> and password file <none>
I20260924 04:45:23.028172 19664 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:23.028260 19664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:23.028505 19664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:23.030834 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a685f4c219b413cbdb65cd7bf33765c"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "4ef2648e368d35d998299bc39544bd8b"
server_key_iv: "2eef2f5e4426a6a8e40821a05fe6a2a8"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.031430 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a685f4c219b413cbdb65cd7bf33765c"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "4ef2648e368d35d998299bc39544bd8b"
server_key_iv: "2eef2f5e4426a6a8e40821a05fe6a2a8"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.035363 19664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:45:23.037931 19679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.039237 19664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:23.039362 19664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "8a685f4c219b413cbdb65cd7bf33765c"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "4ef2648e368d35d998299bc39544bd8b"
server_key_iv: "2eef2f5e4426a6a8e40821a05fe6a2a8"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.039494 19664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:23.055637 19664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:23.058710 19664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:23.058930 19664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:23.059602 19664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:23.060439 19664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:23.060518 19664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.060600 19664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:23.060649 19664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.069634 19664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36041
I20260924 04:45:23.069692 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36041 every 8 connection(s)
I20260924 04:45:23.070598 19664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:45:23.080266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19664
I20260924 04:45:23.080384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
Sep 24 04:45:23 dist-test-slave-npg2 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:23.080727 392 external_mini_cluster.cc:1428] Setting key 64d84ea41ca71ff3b203b1e9bf6e97a1
I20260924 04:45:23.085788 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.072508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60881 (local address 127.0.98.62:38051)
0924 04:45:23.072650 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.072661 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.073484 (+ 823us) server_negotiation.cc:407] Connection header received
0924 04:45:23.074710 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.074713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.074761 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.074837 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:23.076128 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.076706 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.077991 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.078249 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.078341 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.081390 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:23.081393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:23.081396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:23.081422 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:23.083234 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.083814 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.083820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.083825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.083917 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.084497 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.084500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.084501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.084681 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:23.084753 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.085501 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.085664 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:45:23.086724 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.072853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38051 (local address 127.0.98.1:60881)
0924 04:45:23.073296 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.073341 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.074498 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.075004 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.075011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.075451 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:23.075947 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.075961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.076887 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.076890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.077856 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.077865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.078412 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.078415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.078560 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.079370 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:23.079386 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:23.081169 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:23.083405 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:23.083411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:23.083425 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:23.083712 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:23.084151 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:23.084156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:23.084161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:23.084386 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:23.084794 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:23.084802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:23.085200 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.085464 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.085802 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260924 04:45:23.088212 19793 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38051
I20260924 04:45:23.088474 19793 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:23.089079 19793 heartbeater.cc:511] Master 127.0.98.62:38051 requested a full tablet report, sending...
I20260924 04:45:23.090809 19602 ts_manager.cc:194] Registered new tserver with Master: 8a685f4c219b413cbdb65cd7bf33765c (127.0.98.1:36041)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:45:23.092198 19602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:45:23.111105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38051
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:23.224906 19798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:23.225183 19798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:23.225226 19798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:23.229326 19798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:23.229385 19798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:23.229445 19798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:45:23.234653 19798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:23.236022 19798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:23.237108 19798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:23.247211 19804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:23.247207 19803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:23.247442 19798 server_base.cc:1034] running on GCE node
W20260924 04:45:23.247198 19806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:23.248104 19798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:23.248764 19798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:23.249971 19798 hybrid_clock.cc:648] HybridClock initialized: now 1790225123249952 us; error 32 us; skew 500 ppm
Sep 24 04:45:23 dist-test-slave-npg2 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:23.253003 19798 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:45:23.253873 19798 webserver.cc:460] Webserver started at http://127.0.98.2:41663/ using document root <none> and password file <none>
I20260924 04:45:23.254456 19798 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:23.254516 19798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:23.254686 19798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:23.256830 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "030ef2ee95024cc29816e8d6b4d520fd"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "de0227e7de25840c0d2ad48d0733efa5"
server_key_iv: "2e76ce25f690e04c5f84e66ad122d80d"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.257357 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "030ef2ee95024cc29816e8d6b4d520fd"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "de0227e7de25840c0d2ad48d0733efa5"
server_key_iv: "2e76ce25f690e04c5f84e66ad122d80d"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.261395 19798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:23.264209 19813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.265519 19798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:23.265713 19798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "030ef2ee95024cc29816e8d6b4d520fd"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "de0227e7de25840c0d2ad48d0733efa5"
server_key_iv: "2e76ce25f690e04c5f84e66ad122d80d"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.265830 19798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:23.285835 19798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:23.289363 19798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:23.289572 19798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:23.290199 19798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:23.291035 19798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:23.291108 19798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.291194 19798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:23.291237 19798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.301939 19798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32879
I20260924 04:45:23.301960 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32879 every 8 connection(s)
I20260924 04:45:23.302971 19798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:45:23.308524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19798
I20260924 04:45:23.308698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:45:23.309024 392 external_mini_cluster.cc:1428] Setting key f4280dcdf40fae262700fea72d19c58f
Sep 24 04:45:23 dist-test-slave-npg2 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:45:23.319523 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.305016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37703 (local address 127.0.98.62:38051)
0924 04:45:23.305174 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.305178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.306073 (+ 895us) server_negotiation.cc:407] Connection header received
0924 04:45:23.307650 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.307653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.307716 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.307828 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:23.309238 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.309979 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.311819 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.312132 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.312225 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.315284 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:23.315289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:23.315295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:23.315329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:23.317175 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.317768 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.317773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.317775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.317873 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.318224 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.318230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.318232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.318480 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:23.318592 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.319183 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.319307 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:45:23.320381 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.305446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38051 (local address 127.0.98.2:37703)
0924 04:45:23.305913 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.305957 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.307360 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.308014 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.308022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.308432 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:23.309080 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.309102 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.310123 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.310126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.311681 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.311696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.312298 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.312305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.312554 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.313280 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:23.313297 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:23.315020 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:23.317306 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:23.317312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:23.317324 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:23.317635 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:23.317970 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:23.317973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:23.317974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:23.318115 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:23.318615 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:23.318620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:23.318863 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.319031 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.319373 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260924 04:45:23.321760 19927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38051
I20260924 04:45:23.322021 19927 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:45:23.322659 19927 heartbeater.cc:511] Master 127.0.98.62:38051 requested a full tablet report, sending...
I20260924 04:45:23.323940 19602 ts_manager.cc:194] Registered new tserver with Master: 030ef2ee95024cc29816e8d6b4d520fd (127.0.98.2:32879)
I20260924 04:45:23.324545 19602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:45:23.342401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38051
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:23.457139 19932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:23.457419 19932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:23.457472 19932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:23.461620 19932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:23.461689 19932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:23.461753 19932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:45:23.467262 19932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:23.468447 19932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:23.469480 19932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:23.479373 19940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:23.479365 19937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:23.479365 19938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:23.479418 19932 server_base.cc:1034] running on GCE node
I20260924 04:45:23.480146 19932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:23.480798 19932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:23.481990 19932 hybrid_clock.cc:648] HybridClock initialized: now 1790225123481968 us; error 42 us; skew 500 ppm
Sep 24 04:45:23 dist-test-slave-npg2 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:23.484890 19932 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:45:23.485759 19932 webserver.cc:460] Webserver started at http://127.0.98.3:32831/ using document root <none> and password file <none>
I20260924 04:45:23.486330 19932 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:23.486423 19932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:23.486662 19932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:23.488816 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe434631c1914615b891890baf9e630b"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "791ada9b481f02d56afb5e388cbc886c"
server_key_iv: "d486905852f6360c436d753668b80804"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.489364 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe434631c1914615b891890baf9e630b"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "791ada9b481f02d56afb5e388cbc886c"
server_key_iv: "d486905852f6360c436d753668b80804"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.493199 19932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:23.495814 19947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.497287 19932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:23.497454 19932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "fe434631c1914615b891890baf9e630b"
format_stamp: "Formatted at 2026-09-24 04:45:23 on dist-test-slave-npg2"
server_key: "791ada9b481f02d56afb5e388cbc886c"
server_key_iv: "d486905852f6360c436d753668b80804"
server_key_version: "encryptionkey@0"
I20260924 04:45:23.497627 19932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:23.525127 19932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:23.528388 19932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:23.528609 19932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:23.529243 19932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:23.530095 19932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:23.530169 19932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.530234 19932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:23.530284 19932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:23.539479 19932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38677
I20260924 04:45:23.539618 20060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38677 every 8 connection(s)
I20260924 04:45:23.540424 19932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:45:23.549628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 19932
I20260924 04:45:23.549767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:45:23.550063 392 external_mini_cluster.cc:1428] Setting key 5330f0b1623528ff40d17412a696a246
Sep 24 04:45:23 dist-test-slave-npg2 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:23.554782 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.542471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51957 (local address 127.0.98.62:38051)
0924 04:45:23.542615 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.542619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.543523 (+ 904us) server_negotiation.cc:407] Connection header received
0924 04:45:23.544845 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.544847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.544897 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.544976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:23.546163 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.546733 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.547894 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.548111 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.548195 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.551000 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:23.551007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:23.551013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:23.551048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:23.552818 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.553311 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.553320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.553321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.553405 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.553732 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.553734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.553735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.553936 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:23.554056 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.554567 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.554665 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:45:23.555711 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.542793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38051 (local address 127.0.98.3:51957)
0924 04:45:23.543371 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.543415 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.544627 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.545172 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.545179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.545542 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:23.546019 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.546032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.546846 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.546848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.547788 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.547796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.548241 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.548244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.548380 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.548953 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:23.548969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:23.550786 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:23.552932 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:23.552937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:23.552950 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:23.553218 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:23.553494 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:23.553496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:23.553498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:23.553637 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:23.554069 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:23.554073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:23.554348 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.554507 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.554779 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":132,"threads_started":1}
I20260924 04:45:23.556929 20061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38051
I20260924 04:45:23.557157 20061 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:23.557616 20061 heartbeater.cc:511] Master 127.0.98.62:38051 requested a full tablet report, sending...
I20260924 04:45:23.558595 19602 ts_manager.cc:194] Registered new tserver with Master: fe434631c1914615b891890baf9e630b (127.0.98.3:38677)
I20260924 04:45:23.559240 19602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51957
I20260924 04:45:23.564610 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:45:23.574816 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.566704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.0.98.62:38051)
0924 04:45:23.566863 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.566867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.567092 (+ 225us) server_negotiation.cc:407] Connection header received
0924 04:45:23.567240 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.567243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.567287 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.567360 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:23.568085 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.568660 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.569653 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.569860 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.569941 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.571518 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:23.571521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:23.571524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:23.571554 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:23.573164 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.573604 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.573608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.573609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.573693 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:23.574006 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:23.574008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:23.574010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:23.574242 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:23.574352 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.574582 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.574690 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:45:23.578143 19602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:45:23.581450 19602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:45:23.596551 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.590607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32879 (local address 127.0.0.1:57374)
0924 04:45:23.591555 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.591615 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.591746 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.592414 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.592421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.592444 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:23.592609 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.592615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.593761 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.593767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.594914 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.594925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.596006 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.596009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.596120 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.596156 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.596271 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.596391 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":96,"threads_started":1}
I20260924 04:45:23.596565 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.590740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.98.2:32879)
0924 04:45:23.591877 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.591890 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.591914 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:45:23.591993 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.591999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.592232 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.592414 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:23.592723 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.593637 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.595054 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.595887 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.596006 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.596157 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.596305 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.596392 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":102,"threads_started":1}
I20260924 04:45:23.597170 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.590640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38677 (local address 127.0.0.1:60644)
0924 04:45:23.591555 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.591615 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.592422 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.592906 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.592909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.592928 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:23.593018 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.593023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.594397 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.594400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.595469 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.595478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.596769 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.596772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.596853 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.596873 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.596952 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.597020 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260924 04:45:23.597241 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.591495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.0.98.3:38677)
0924 04:45:23.592188 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.592198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.592218 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:45:23.592583 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.592588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.592764 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.592916 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:23.593129 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.594237 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.595595 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.596412 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.596543 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.596682 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.597013 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.597104 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":238,"threads_started":1}
I20260924 04:45:23.597873 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.591295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.98.1:36041)
0924 04:45:23.592003 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.592008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.592029 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:45:23.592213 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.592218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.592428 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.592575 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:23.592995 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.594230 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.595733 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.596467 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.596559 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.597586 (+ 1027us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.597640 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.597729 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":105,"threads_started":1}
I20260924 04:45:23.599388 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.590936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36041 (local address 127.0.0.1:59050)
0924 04:45:23.591967 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.591979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.592055 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.592709 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.592712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.592727 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:23.592866 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.592876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.594355 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.594359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.595608 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.595618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.596997 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.597001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.597088 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.597109 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.597171 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.599296 (+ 2125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":54,"thread_start_us":85,"threads_started":1}
I20260924 04:45:23.599900 19995 tablet_service.cc:1467] Processing CreateTablet for tablet 88fc2a94c80d4b37946b074e805c8ff7 (DEFAULT_TABLE table=test-table [id=2ede9fb646c64963846f2075b82bfe11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:23.599897 19861 tablet_service.cc:1467] Processing CreateTablet for tablet 88fc2a94c80d4b37946b074e805c8ff7 (DEFAULT_TABLE table=test-table [id=2ede9fb646c64963846f2075b82bfe11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:23.601068 19861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fc2a94c80d4b37946b074e805c8ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:23.601068 19995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fc2a94c80d4b37946b074e805c8ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:23.608819 20079 tablet_bootstrap.cc:492] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Bootstrap starting.
I20260924 04:45:23.609589 20078 tablet_bootstrap.cc:492] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Bootstrap starting.
I20260924 04:45:23.609674 19727 tablet_service.cc:1467] Processing CreateTablet for tablet 88fc2a94c80d4b37946b074e805c8ff7 (DEFAULT_TABLE table=test-table [id=2ede9fb646c64963846f2075b82bfe11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:23.610867 19727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fc2a94c80d4b37946b074e805c8ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:23.612011 20078 tablet_bootstrap.cc:654] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:23.613333 20078 log.cc:824] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:23.613690 20079 tablet_bootstrap.cc:654] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:23.614507 20079 log.cc:824] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:23.615752 20078 tablet_bootstrap.cc:492] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: No bootstrap required, opened a new log
I20260924 04:45:23.615952 20078 ts_tablet_manager.cc:1397] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:45:23.616354 20079 tablet_bootstrap.cc:492] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: No bootstrap required, opened a new log
I20260924 04:45:23.616601 20079 ts_tablet_manager.cc:1397] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 04:45:23.618609 20082 tablet_bootstrap.cc:492] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: Bootstrap starting.
I20260924 04:45:23.620061 20078 raft_consensus.cc:348] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.620038 20079 raft_consensus.cc:348] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.620325 20079 raft_consensus.cc:374] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:23.620327 20078 raft_consensus.cc:374] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:23.620383 20079 raft_consensus.cc:729] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030ef2ee95024cc29816e8d6b4d520fd, State: Initialized, Role: FOLLOWER
I20260924 04:45:23.620390 20078 raft_consensus.cc:729] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe434631c1914615b891890baf9e630b, State: Initialized, Role: FOLLOWER
I20260924 04:45:23.620816 20078 consensus_queue.cc:260] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.620827 20079 consensus_queue.cc:260] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.621397 20082 tablet_bootstrap.cc:654] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:23.621476 19927 heartbeater.cc:503] Master 127.0.98.62:38051 was elected leader, sending a full tablet report...
I20260924 04:45:23.621799 20078 ts_tablet_manager.cc:1428] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:45:23.621816 20079 ts_tablet_manager.cc:1428] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:45:23.622203 20061 heartbeater.cc:503] Master 127.0.98.62:38051 was elected leader, sending a full tablet report...
I20260924 04:45:23.622233 20082 log.cc:824] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:23.624362 20082 tablet_bootstrap.cc:492] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: No bootstrap required, opened a new log
I20260924 04:45:23.624579 20082 ts_tablet_manager.cc:1397] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:45:23.628594 20082 raft_consensus.cc:348] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.628844 20082 raft_consensus.cc:374] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:23.628922 20082 raft_consensus.cc:729] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a685f4c219b413cbdb65cd7bf33765c, State: Initialized, Role: FOLLOWER
I20260924 04:45:23.629352 20082 consensus_queue.cc:260] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.630015 19793 heartbeater.cc:503] Master 127.0.98.62:38051 was elected leader, sending a full tablet report...
I20260924 04:45:23.630349 20082 ts_tablet_manager.cc:1428] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 04:45:23.791478 20062 tablet.cc:2369] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:45:23.804138 19928 tablet.cc:2369] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:45:23.822068 19794 tablet.cc:2369] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:23.840694 20084 raft_consensus.cc:482] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:23.840900 20084 raft_consensus.cc:504] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.842098 20084 leader_election.cc:288] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 030ef2ee95024cc29816e8d6b4d520fd (127.0.98.2:32879), 8a685f4c219b413cbdb65cd7bf33765c (127.0.98.1:36041)
I20260924 04:45:23.846213 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.842487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57961 (local address 127.0.98.2:32879)
0924 04:45:23.842631 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.842638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.842878 (+ 240us) server_negotiation.cc:407] Connection header received
0924 04:45:23.843021 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.843025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.843082 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.843231 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:23.843604 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.844205 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.845257 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.845633 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.845746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.845814 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.846005 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.846084 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:45:23.846213 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.842412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36041 (local address 127.0.98.3:58807)
0924 04:45:23.842619 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.842636 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.842739 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.843059 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.843062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.843080 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:23.843237 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.843243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.844149 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.844152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.845058 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.845068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.845785 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.845788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.845904 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.845920 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.846019 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.846085 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 04:45:23.846239 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.842532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58807 (local address 127.0.98.1:36041)
0924 04:45:23.842675 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.842680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.842697 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:45:23.842882 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.842885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.842945 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.843034 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:23.843345 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.844021 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.845191 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.845640 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.845761 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.845825 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.846053 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.846131 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:45:23.846223 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.842444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32879 (local address 127.0.98.3:57961)
0924 04:45:23.842796 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.842811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:23.842902 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:23.843241 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:23.843244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:23.843261 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:23.843385 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.843390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.844314 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.844317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.845093 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:23.845110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.845752 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.845756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:23.845856 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.845878 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:45:23.845949 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:45:23.846041 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260924 04:45:23.846881 19881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc2a94c80d4b37946b074e805c8ff7" candidate_uuid: "fe434631c1914615b891890baf9e630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030ef2ee95024cc29816e8d6b4d520fd" is_pre_election: true
I20260924 04:45:23.846884 19747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc2a94c80d4b37946b074e805c8ff7" candidate_uuid: "fe434631c1914615b891890baf9e630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a685f4c219b413cbdb65cd7bf33765c" is_pre_election: true
I20260924 04:45:23.847226 19881 raft_consensus.cc:2454] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe434631c1914615b891890baf9e630b in term 0.
I20260924 04:45:23.847244 19747 raft_consensus.cc:2454] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe434631c1914615b891890baf9e630b in term 0.
I20260924 04:45:23.847718 19951 leader_election.cc:302] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 030ef2ee95024cc29816e8d6b4d520fd, 8a685f4c219b413cbdb65cd7bf33765c, fe434631c1914615b891890baf9e630b; no voters:
I20260924 04:45:23.848021 20084 raft_consensus.cc:2781] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:45:23.848107 20084 raft_consensus.cc:482] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:45:23.848138 20084 raft_consensus.cc:3037] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:23.849202 20084 raft_consensus.cc:504] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.849591 20084 leader_election.cc:288] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [CANDIDATE]: Term 1 election: Requested vote from peers 030ef2ee95024cc29816e8d6b4d520fd (127.0.98.2:32879), 8a685f4c219b413cbdb65cd7bf33765c (127.0.98.1:36041)
I20260924 04:45:23.849920 19881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc2a94c80d4b37946b074e805c8ff7" candidate_uuid: "fe434631c1914615b891890baf9e630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030ef2ee95024cc29816e8d6b4d520fd"
I20260924 04:45:23.849957 19747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc2a94c80d4b37946b074e805c8ff7" candidate_uuid: "fe434631c1914615b891890baf9e630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a685f4c219b413cbdb65cd7bf33765c"
I20260924 04:45:23.850052 19881 raft_consensus.cc:3037] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:23.850080 19747 raft_consensus.cc:3037] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:23.851069 19881 raft_consensus.cc:2454] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe434631c1914615b891890baf9e630b in term 1.
I20260924 04:45:23.851179 19747 raft_consensus.cc:2454] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe434631c1914615b891890baf9e630b in term 1.
I20260924 04:45:23.851420 19951 leader_election.cc:302] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030ef2ee95024cc29816e8d6b4d520fd, fe434631c1914615b891890baf9e630b; no voters:
I20260924 04:45:23.851648 20084 raft_consensus.cc:2781] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:23.851917 20084 raft_consensus.cc:686] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 1 LEADER]: Becoming Leader. State: Replica: fe434631c1914615b891890baf9e630b, State: Running, Role: LEADER
I20260924 04:45:23.852253 20084 consensus_queue.cc:237] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } }
I20260924 04:45:23.855767 19601 catalog_manager.cc:5495] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b reported cstate change: term changed from 0 to 1, leader changed from <none> to fe434631c1914615b891890baf9e630b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fe434631c1914615b891890baf9e630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe434631c1914615b891890baf9e630b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 } health_report { overall_health: UNKNOWN } } }
I20260924 04:45:23.890571 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.886524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.98.3:38677)
0924 04:45:23.886687 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:23.886691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:23.886706 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:45:23.886783 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:23.886786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:23.886839 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:23.886960 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:23.887226 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.887793 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.888649 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:23.888930 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:23.889091 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:23.889487 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:23.890149 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:23.890247 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:23.890320 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:45:23.890371 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:45:23.896476 19747 raft_consensus.cc:1261] T 88fc2a94c80d4b37946b074e805c8ff7 P 8a685f4c219b413cbdb65cd7bf33765c [term 1 FOLLOWER]: Refusing update from remote peer fe434631c1914615b891890baf9e630b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:23.896492 19881 raft_consensus.cc:1261] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 1 FOLLOWER]: Refusing update from remote peer fe434631c1914615b891890baf9e630b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:23.897149 20084 consensus_queue.cc:1035] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [LEADER]: Connected to new peer: Peer: permanent_uuid: "030ef2ee95024cc29816e8d6b4d520fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:23.897356 20088 consensus_queue.cc:1035] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a685f4c219b413cbdb65cd7bf33765c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:23.913301 19602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:23.913516 19602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:23.916114 19602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae82326f603c484cb01b5fd532768647: Sending DeleteTablet for 3 replicas of tablet 88fc2a94c80d4b37946b074e805c8ff7
I20260924 04:45:23.917033 19861 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fc2a94c80d4b37946b074e805c8ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57374
I20260924 04:45:23.917057 19727 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fc2a94c80d4b37946b074e805c8ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260924 04:45:23.917559 20101 tablet_replica.cc:330] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: stopping tablet replica
I20260924 04:45:23.917551 19995 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fc2a94c80d4b37946b074e805c8ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60644
I20260924 04:45:23.917973 20101 raft_consensus.cc:2229] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:23.918332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19664
I20260924 04:45:23.918334 20101 raft_consensus.cc:2258] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:23.918902 20103 tablet_replica.cc:330] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: stopping tablet replica
I20260924 04:45:23.919238 20103 raft_consensus.cc:2229] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:45:23.919674 20103 raft_consensus.cc:2258] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:23.921293 20101 ts_tablet_manager.cc:1905] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:23.924161 20103 ts_tablet_manager.cc:1905] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:23.924893 20101 ts_tablet_manager.cc:1918] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:23.925005 20101 log.cc:1196] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-1/wal/wals/88fc2a94c80d4b37946b074e805c8ff7
I20260924 04:45:23.925359 20101 ts_tablet_manager.cc:1939] T 88fc2a94c80d4b37946b074e805c8ff7 P 030ef2ee95024cc29816e8d6b4d520fd: Deleting consensus metadata
I20260924 04:45:23.926805 19589 catalog_manager.cc:4841] TS 030ef2ee95024cc29816e8d6b4d520fd (127.0.98.2:32879): tablet 88fc2a94c80d4b37946b074e805c8ff7 (table test-table [id=2ede9fb646c64963846f2075b82bfe11]) successfully deleted
I20260924 04:45:23.927186 20103 ts_tablet_manager.cc:1918] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:23.927286 20103 log.cc:1196] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/88fc2a94c80d4b37946b074e805c8ff7
I20260924 04:45:23.927615 20103 ts_tablet_manager.cc:1939] T 88fc2a94c80d4b37946b074e805c8ff7 P fe434631c1914615b891890baf9e630b: Deleting consensus metadata
W20260924 04:45:23.928234 19587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36041 (error 108)
I20260924 04:45:23.928752 19588 catalog_manager.cc:4841] TS fe434631c1914615b891890baf9e630b (127.0.98.3:38677): tablet 88fc2a94c80d4b37946b074e805c8ff7 (table test-table [id=2ede9fb646c64963846f2075b82bfe11]) successfully deleted
I20260924 04:45:23.929257 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:23.928977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36041 (local address 127.0.0.1:59064)
0924 04:45:23.929091 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:23.929172 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:45:23.929382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19798
W20260924 04:45:23.929482 19587 catalog_manager.cc:4584] TS 8a685f4c219b413cbdb65cd7bf33765c (127.0.98.1:36041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88fc2a94c80d4b37946b074e805c8ff7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36041: connect: Connection refused (error 111)
I20260924 04:45:23.936697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19932
I20260924 04:45:23.942680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 19570
2026-09-24T04:45:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2565 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T04:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:23Z Disabled control of system clock
I20260924 04:45:23.973618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34763
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:45:24.086541 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:24.086824 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:24.086860 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:24.090907 20108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:24.090973 20108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:24.090989 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:24.091006 20108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:24.091022 20108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:24.095860 20108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:24.096998 20108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:24.098381 20108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:24.107620 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.107666 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:24.107661 20108 server_base.cc:1034] running on GCE node
W20260924 04:45:24.107615 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:24.108431 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:24.109822 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:24.111023 20108 hybrid_clock.cc:648] HybridClock initialized: now 1790225124110983 us; error 54 us; skew 500 ppm
I20260924 04:45:24.113107 20108 webserver.cc:460] Webserver started at http://127.0.98.62:42681/ using document root <none> and password file <none>
I20260924 04:45:24.113725 20108 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:24.113821 20108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:24.114058 20108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:24.116237 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "a2b6210b6eab46e9a054e41617bf8ff6"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "573b491c660e2d9bf7f51d4fc03eafbf"
server_key_iv: "86b9d7679cae329fb529f2c92218643f"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.116819 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2b6210b6eab46e9a054e41617bf8ff6"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "573b491c660e2d9bf7f51d4fc03eafbf"
server_key_iv: "86b9d7679cae329fb529f2c92218643f"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.120769 20108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:45:24.123370 20122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.124737 20108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:24.124884 20108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "a2b6210b6eab46e9a054e41617bf8ff6"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "573b491c660e2d9bf7f51d4fc03eafbf"
server_key_iv: "86b9d7679cae329fb529f2c92218643f"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.125002 20108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:24.142835 20108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:24.143637 20108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:24.143843 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:24.151207 20108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34763
I20260924 04:45:24.151229 20174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34763 every 8 connection(s)
I20260924 04:45:24.152194 20108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:24.155507 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:24.159746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20108
I20260924 04:45:24.159875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:24.160179 392 external_mini_cluster.cc:1428] Setting key 7d1163364c2407b1dddf3765ea148595
I20260924 04:45:24.161710 20175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6: Bootstrap starting.
I20260924 04:45:24.164177 20175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:24.164995 20175 log.cc:824] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:24.166852 20175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6: No bootstrap required, opened a new log
I20260924 04:45:24.169561 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.161738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.0.98.62:34763)
0924 04:45:24.162425 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.162441 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.162478 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:45:24.163152 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.163171 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.163580 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.163870 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:24.164287 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.165368 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.166312 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.166787 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.166967 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.167842 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:24.167847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:24.167859 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:24.168257 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:24.168310 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:24.168329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:24.168436 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.168614 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.168860 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260924 04:45:24.170807 20175 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34763 } }
I20260924 04:45:24.171042 20175 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:24.171121 20175 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b6210b6eab46e9a054e41617bf8ff6, State: Initialized, Role: FOLLOWER
I20260924 04:45:24.171605 20175 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34763 } }
I20260924 04:45:24.171765 20175 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:24.171824 20175 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:24.171916 20175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:24.172827 20175 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34763 } }
I20260924 04:45:24.173231 20175 leader_election.cc:302] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2b6210b6eab46e9a054e41617bf8ff6; no voters:
I20260924 04:45:24.173771 20175 leader_election.cc:288] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:24.173944 20180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:24.174240 20180 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: a2b6210b6eab46e9a054e41617bf8ff6, State: Running, Role: LEADER
I20260924 04:45:24.174557 20180 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34763 } }
I20260924 04:45:24.175230 20175 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:24.175699 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2b6210b6eab46e9a054e41617bf8ff6. Latest consensus state: current_term: 1 leader_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34763 } } }
I20260924 04:45:24.175806 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:24.176424 20181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b6210b6eab46e9a054e41617bf8ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34763 } } }
I20260924 04:45:24.176522 20181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:24.176810 20184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:24.180619 20184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:24.186546 20184 catalog_manager.cc:1347] Generated new cluster ID: 0379314b1e014b60abef80c02c232e76
I20260924 04:45:24.186623 20184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:24.193277 20184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:24.194475 20184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:24.204716 20184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6: Generated new TSK 0
I20260924 04:45:24.205439 20184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:24.216427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34763
--builtin_ntp_servers=127.0.98.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:45:24.330791 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:24.331070 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:24.331110 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:24.335319 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:24.335382 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:24.335440 20199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:45:24.340317 20199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:24.341449 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:24.342854 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:24.352955 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.352975 20205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.352953 20204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:24.353130 20199 server_base.cc:1034] running on GCE node
I20260924 04:45:24.353695 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:24.354442 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:24.355652 20199 hybrid_clock.cc:648] HybridClock initialized: now 1790225124355644 us; error 31 us; skew 500 ppm
I20260924 04:45:24.357718 20199 webserver.cc:460] Webserver started at http://127.0.98.1:33419/ using document root <none> and password file <none>
I20260924 04:45:24.358340 20199 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:24.358453 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:24.358798 20199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:24.361009 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data/instance:
uuid: "c49a76bd968b4e859b2e857bc16b20ff"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "bcf0438db2f02d4338f30aa1a3dd7f34"
server_key_iv: "f0487014bcda20fb7d224333576486cc"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.361622 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c49a76bd968b4e859b2e857bc16b20ff"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "bcf0438db2f02d4338f30aa1a3dd7f34"
server_key_iv: "f0487014bcda20fb7d224333576486cc"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.365720 20199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:24.368440 20213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.369877 20199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:45:24.370122 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal
uuid: "c49a76bd968b4e859b2e857bc16b20ff"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "bcf0438db2f02d4338f30aa1a3dd7f34"
server_key_iv: "f0487014bcda20fb7d224333576486cc"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.370256 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:24.427055 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:24.427860 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:24.428071 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:24.428778 20199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:24.429977 20199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:24.430052 20199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.430122 20199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:24.430172 20199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.439313 20199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44555
I20260924 04:45:24.439337 20326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44555 every 8 connection(s)
I20260924 04:45:24.440253 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:45:24.444265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20199
I20260924 04:45:24.444418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:45:24.444777 392 external_mini_cluster.cc:1428] Setting key 96da69a798da076912d9208b89f7551e
I20260924 04:45:24.447206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34763
--builtin_ntp_servers=127.0.98.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:45:24.451709 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.442192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55013 (local address 127.0.98.62:34763)
0924 04:45:24.442327 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.442331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.443603 (+ 1272us) server_negotiation.cc:407] Connection header received
0924 04:45:24.445186 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.445189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.445248 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.445361 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:24.446692 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.447541 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.449305 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.449691 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.449814 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.451005 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:24.451012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:24.451018 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:24.451037 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:24.451065 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:24.451078 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:24.451191 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.451418 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.451496 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:45:24.452451 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.442730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34763 (local address 127.0.98.1:55013)
0924 04:45:24.443449 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:24.443490 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:24.444965 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:24.445569 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:24.445576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:24.445698 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:24.446463 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.446482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.447713 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.447720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.449165 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.449175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.449872 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.449876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.450049 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.450721 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:24.450722 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:24.450767 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:24.450834 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:24.451233 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:24.451244 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:24.451252 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:45:24.451387 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 04:45:24.451628 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260924 04:45:24.453860 20327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34763
I20260924 04:45:24.454120 20327 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:24.454702 20327 heartbeater.cc:511] Master 127.0.98.62:34763 requested a full tablet report, sending...
I20260924 04:45:24.456347 20139 ts_manager.cc:194] Registered new tserver with Master: c49a76bd968b4e859b2e857bc16b20ff (127.0.98.1:44555)
I20260924 04:45:24.457497 20139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55013
W20260924 04:45:24.562453 20330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:24.562793 20330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:24.562865 20330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:24.567006 20330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:24.567107 20330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:24.567219 20330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:45:24.572116 20330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:24.573282 20330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:24.574896 20330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:24.585212 20335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.585222 20336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.585229 20338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:24.585276 20330 server_base.cc:1034] running on GCE node
I20260924 04:45:24.585963 20330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:24.586772 20330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:24.587950 20330 hybrid_clock.cc:648] HybridClock initialized: now 1790225124587946 us; error 64 us; skew 500 ppm
I20260924 04:45:24.589954 20330 webserver.cc:460] Webserver started at http://127.0.98.2:46721/ using document root <none> and password file <none>
I20260924 04:45:24.590582 20330 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:24.590667 20330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:24.590893 20330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:24.593050 20330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "23c1e89e693b207dd8cbd14df02df438"
server_key_iv: "32c370ab7f4792a370410472f0bb3427"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.593628 20330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "23c1e89e693b207dd8cbd14df02df438"
server_key_iv: "32c370ab7f4792a370410472f0bb3427"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.597438 20330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:24.599979 20344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.601202 20330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:24.601445 20330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal
uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "23c1e89e693b207dd8cbd14df02df438"
server_key_iv: "32c370ab7f4792a370410472f0bb3427"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.601567 20330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:24.618531 20330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:24.619372 20330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:24.619609 20330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:24.620242 20330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:24.621219 20330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:24.621295 20330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.621363 20330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:24.621412 20330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.630545 20330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39933
I20260924 04:45:24.630563 20457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39933 every 8 connection(s)
I20260924 04:45:24.631541 20330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:45:24.634661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20330
I20260924 04:45:24.634856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:45:24.635339 392 external_mini_cluster.cc:1428] Setting key 09ebc2b443110a57f2e1fb67da07de12
I20260924 04:45:24.638917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34763
--builtin_ntp_servers=127.0.98.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:45:24.645051 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.633812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40741 (local address 127.0.98.62:34763)
0924 04:45:24.634000 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.634006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.636687 (+ 2681us) server_negotiation.cc:407] Connection header received
0924 04:45:24.638646 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.638652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.638715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.638814 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:24.640005 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.641046 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.642549 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.642966 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.643095 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.644379 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:24.644385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:24.644390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:24.644410 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:24.644441 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:24.644456 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:24.644553 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.644730 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.644814 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:45:24.645815 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.634407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34763 (local address 127.0.98.2:40741)
0924 04:45:24.636507 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:24.636565 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:24.638384 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:24.639036 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:24.639047 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:24.639195 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:24.639853 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.639867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.641207 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.641211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.642410 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.642420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.643209 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.643220 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.643370 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.644066 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:24.644068 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:24.644121 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:24.644189 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:24.644565 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:24.644571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:24.644575 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:45:24.644668 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:45:24.644907 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1571,"thread_start_us":154,"threads_started":1}
I20260924 04:45:24.647043 20458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34763
I20260924 04:45:24.647357 20458 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:24.647897 20458 heartbeater.cc:511] Master 127.0.98.62:34763 requested a full tablet report, sending...
I20260924 04:45:24.648988 20139 ts_manager.cc:194] Registered new tserver with Master: 9e8a40a6c2f046fa9d789c7f8fc8e3b4 (127.0.98.2:39933)
I20260924 04:45:24.649572 20139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40741
W20260924 04:45:24.755288 20461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:24.755551 20461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:24.755590 20461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:24.759790 20461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:24.759850 20461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:24.759953 20461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:45:24.764789 20461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:24.766022 20461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:24.767475 20461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:24.777786 20467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.777761 20469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:24.777829 20466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:24.778148 20461 server_base.cc:1034] running on GCE node
I20260924 04:45:24.778656 20461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:24.779408 20461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:24.780596 20461 hybrid_clock.cc:648] HybridClock initialized: now 1790225124780578 us; error 31 us; skew 500 ppm
I20260924 04:45:24.782711 20461 webserver.cc:460] Webserver started at http://127.0.98.3:44001/ using document root <none> and password file <none>
I20260924 04:45:24.783401 20461 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:24.783465 20461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:24.783694 20461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:24.785751 20461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "a3b5f3703f3e47df14ba501dba84e1ea"
server_key_iv: "0edb2796a35ced19a369459ac21ad694"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.786302 20461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "a3b5f3703f3e47df14ba501dba84e1ea"
server_key_iv: "0edb2796a35ced19a369459ac21ad694"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.790232 20461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:24.792892 20475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.794366 20461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:24.794512 20461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal
uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3"
format_stamp: "Formatted at 2026-09-24 04:45:24 on dist-test-slave-npg2"
server_key: "a3b5f3703f3e47df14ba501dba84e1ea"
server_key_iv: "0edb2796a35ced19a369459ac21ad694"
server_key_version: "encryptionkey@0"
I20260924 04:45:24.794636 20461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:24.821745 20461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:24.822523 20461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:24.822743 20461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:24.823390 20461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:45:24.824434 20461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:45:24.824509 20461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.824581 20461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:45:24.824633 20461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:24.833895 20461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37725
I20260924 04:45:24.833912 20588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37725 every 8 connection(s)
I20260924 04:45:24.834844 20461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:45:24.836398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20461
I20260924 04:45:24.836534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:45:24.836809 392 external_mini_cluster.cc:1428] Setting key 899fd95a15146df53e907a3790aecbc0
I20260924 04:45:24.848224 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.836968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46025 (local address 127.0.98.62:34763)
0924 04:45:24.837137 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.837142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.838706 (+ 1564us) server_negotiation.cc:407] Connection header received
0924 04:45:24.840727 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.840733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.840805 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.840902 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:24.842425 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.843352 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.845490 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.845921 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.846033 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.847557 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:24.847561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:24.847564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:24.847582 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:24.847685 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:24.847701 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:24.847818 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.848018 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.848095 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:45:24.849218 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.837582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34763 (local address 127.0.98.3:46025)
0924 04:45:24.838494 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:24.838553 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:24.840474 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:24.841168 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:24.841177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:24.841339 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:24.842242 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.842261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.843524 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.843527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.845332 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.845343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.846110 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.846117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.846355 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.847182 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:45:24.847186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:24.847263 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:45:24.847371 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:24.847832 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:24.847842 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:24.847848 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:45:24.848003 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:45:24.848280 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":163,"threads_started":1}
I20260924 04:45:24.850703 20589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34763
I20260924 04:45:24.850973 20589 heartbeater.cc:464] Registering TS with master...
I20260924 04:45:24.851562 20589 heartbeater.cc:511] Master 127.0.98.62:34763 requested a full tablet report, sending...
I20260924 04:45:24.852716 20138 ts_manager.cc:194] Registered new tserver with Master: f5aa05d34ac54a1eb05744cb36bfcbb3 (127.0.98.3:37725)
I20260924 04:45:24.853334 20138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46025
I20260924 04:45:24.863224 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:45:24.869545 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.865276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.0.98.62:34763)
0924 04:45:24.865456 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.865460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.865677 (+ 217us) server_negotiation.cc:407] Connection header received
0924 04:45:24.865850 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.865854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.865909 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.865996 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:24.866287 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.866851 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.867911 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.868161 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.868274 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.869110 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:24.869114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:24.869117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:24.869130 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:24.869157 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:24.869166 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:24.869245 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.869335 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.869394 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 04:45:24.873045 20138 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:45:24.876509 20138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:45:24.892056 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.885576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39933 (local address 127.0.0.1:36574)
0924 04:45:24.886040 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:24.886090 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:24.886274 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:24.887657 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:24.887660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:24.887673 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:24.887745 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.887752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.889113 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.889116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.890399 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.890409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.891511 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.891515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.891634 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.891673 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:45:24.891765 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:45:24.891889 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":57,"thread_start_us":96,"threads_started":1}
I20260924 04:45:24.892078 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.885796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37725 (local address 127.0.0.1:44600)
0924 04:45:24.886684 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:24.886700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:24.886787 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:24.887591 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:24.887594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:24.887645 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:24.887753 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.887758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.888799 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.888805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.890359 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.890370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.891758 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.891762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.891837 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.891856 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:45:24.891930 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:45:24.891984 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260924 04:45:24.892508 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.886272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.0.98.3:37725)
0924 04:45:24.887019 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.887041 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.887061 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:45:24.887152 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.887157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.887302 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.887435 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:24.887898 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.888655 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.890487 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.891371 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.891488 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.891694 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.891999 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.892329 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":101,"threads_started":1}
I20260924 04:45:24.892774 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.886343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44555 (local address 127.0.0.1:50360)
0924 04:45:24.886761 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:24.886771 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:24.886842 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:24.887432 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:24.887438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:24.887460 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:24.887636 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.887644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.890329 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.890332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.891336 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:24.891357 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.892286 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.892291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:24.892408 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.892434 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:45:24.892485 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:45:24.892653 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260924 04:45:24.893134 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.885735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.98.2:39933)
0924 04:45:24.887186 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.887204 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.887223 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:45:24.887285 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.887289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.887469 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.887656 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:24.887863 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.888995 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.890552 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.891370 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.892616 (+ 1246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.892814 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.892864 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.892973 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":148,"threads_started":1}
I20260924 04:45:24.893594 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:24.886505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.0.98.1:44555)
0924 04:45:24.886869 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:24.886879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:24.886911 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:45:24.886992 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:24.886998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:24.887262 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:24.887435 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:24.887769 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.888987 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.891487 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:24.892126 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:24.893064 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:24.893270 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:24.893317 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 04:45:24.893413 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260924 04:45:24.895921 20261 tablet_service.cc:1467] Processing CreateTablet for tablet 78bdf8c1bbe445a88d7365989ea96be4 (DEFAULT_TABLE table=test-table [id=9545f417da8c49d2a9a624ba832c507f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:24.895941 20392 tablet_service.cc:1467] Processing CreateTablet for tablet 78bdf8c1bbe445a88d7365989ea96be4 (DEFAULT_TABLE table=test-table [id=9545f417da8c49d2a9a624ba832c507f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:24.895941 20523 tablet_service.cc:1467] Processing CreateTablet for tablet 78bdf8c1bbe445a88d7365989ea96be4 (DEFAULT_TABLE table=test-table [id=9545f417da8c49d2a9a624ba832c507f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:45:24.897008 20392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bdf8c1bbe445a88d7365989ea96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:24.897008 20523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bdf8c1bbe445a88d7365989ea96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:24.897012 20261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bdf8c1bbe445a88d7365989ea96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:24.903447 20606 tablet_bootstrap.cc:492] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: Bootstrap starting.
I20260924 04:45:24.904636 20608 tablet_bootstrap.cc:492] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Bootstrap starting.
I20260924 04:45:24.905653 20607 tablet_bootstrap.cc:492] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Bootstrap starting.
I20260924 04:45:24.905903 20606 tablet_bootstrap.cc:654] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:24.906774 20608 tablet_bootstrap.cc:654] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:24.906801 20606 log.cc:824] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:24.907615 20608 log.cc:824] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:24.907689 20607 tablet_bootstrap.cc:654] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:24.908522 20607 log.cc:824] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:24.909044 20606 tablet_bootstrap.cc:492] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: No bootstrap required, opened a new log
I20260924 04:45:24.909245 20606 ts_tablet_manager.cc:1397] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:45:24.909595 20608 tablet_bootstrap.cc:492] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: No bootstrap required, opened a new log
I20260924 04:45:24.909781 20608 ts_tablet_manager.cc:1397] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:45:24.910288 20607 tablet_bootstrap.cc:492] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: No bootstrap required, opened a new log
I20260924 04:45:24.910475 20607 ts_tablet_manager.cc:1397] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:45:24.913671 20607 raft_consensus.cc:348] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:24.913950 20607 raft_consensus.cc:374] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:24.914017 20607 raft_consensus.cc:729] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8a40a6c2f046fa9d789c7f8fc8e3b4, State: Initialized, Role: FOLLOWER
I20260924 04:45:24.913847 20606 raft_consensus.cc:348] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:24.914101 20606 raft_consensus.cc:374] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:24.914148 20606 raft_consensus.cc:729] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49a76bd968b4e859b2e857bc16b20ff, State: Initialized, Role: FOLLOWER
I20260924 04:45:24.914044 20608 raft_consensus.cc:348] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:24.914273 20608 raft_consensus.cc:374] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:24.914314 20608 raft_consensus.cc:729] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5aa05d34ac54a1eb05744cb36bfcbb3, State: Initialized, Role: FOLLOWER
I20260924 04:45:24.914459 20607 consensus_queue.cc:260] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:24.914688 20608 consensus_queue.cc:260] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:24.914721 20606 consensus_queue.cc:260] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:24.915462 20458 heartbeater.cc:503] Master 127.0.98.62:34763 was elected leader, sending a full tablet report...
I20260924 04:45:24.915577 20607 ts_tablet_manager.cc:1428] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:45:24.915642 20608 ts_tablet_manager.cc:1428] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:45:24.915719 20606 ts_tablet_manager.cc:1428] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260924 04:45:24.915970 20589 heartbeater.cc:503] Master 127.0.98.62:34763 was elected leader, sending a full tablet report...
I20260924 04:45:24.915970 20327 heartbeater.cc:503] Master 127.0.98.62:34763 was elected leader, sending a full tablet report...
W20260924 04:45:24.941535 20328 tablet.cc:2369] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:45:25.085914 20590 tablet.cc:2369] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:25.086306 20614 raft_consensus.cc:482] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:45:25.086486 20614 raft_consensus.cc:504] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:25.087639 20614 leader_election.cc:288] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e8a40a6c2f046fa9d789c7f8fc8e3b4 (127.0.98.2:39933), c49a76bd968b4e859b2e857bc16b20ff (127.0.98.1:44555)
I20260924 04:45:25.091576 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.087966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39933 (local address 127.0.98.3:58169)
0924 04:45:25.088153 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:25.088171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:25.088288 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:25.088733 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:25.088736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:25.088755 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:25.088861 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:25.088867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.089634 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.089638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:25.090435 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:25.090444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.091115 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.091145 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:25.091248 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:25.091274 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:45:25.091356 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:45:25.091429 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 04:45:25.091593 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.088059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58169 (local address 127.0.98.2:39933)
0924 04:45:25.088200 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:25.088206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:25.088230 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:45:25.088454 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:25.088460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:25.088521 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:25.088633 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:25.088978 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.089511 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.090585 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.090991 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.091152 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:25.091219 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:25.091374 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 04:45:25.091462 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:45:25.092010 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.088292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44555 (local address 127.0.98.3:59089)
0924 04:45:25.088718 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:25.088729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:25.088845 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:25.089112 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:25.089115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:25.089130 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:45:25.089263 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:25.089268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.089986 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.089990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:25.090746 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:25.090754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.091528 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.091531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:25.091640 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:25.091667 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:45:25.091820 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:45:25.091887 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":141,"threads_started":1}
I20260924 04:45:25.092049 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.088441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59089 (local address 127.0.98.1:44555)
0924 04:45:25.088585 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:25.088591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:25.088810 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:45:25.088964 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:25.088967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:25.089014 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:25.089102 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:45:25.089373 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.089865 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.090874 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.091407 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.091527 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:25.091581 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:25.091867 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 04:45:25.091938 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:45:25.092263 20412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bdf8c1bbe445a88d7365989ea96be4" candidate_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" is_pre_election: true
I20260924 04:45:25.092607 20412 raft_consensus.cc:2454] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5aa05d34ac54a1eb05744cb36bfcbb3 in term 0.
I20260924 04:45:25.092770 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bdf8c1bbe445a88d7365989ea96be4" candidate_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49a76bd968b4e859b2e857bc16b20ff" is_pre_election: true
I20260924 04:45:25.093055 20281 raft_consensus.cc:2454] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5aa05d34ac54a1eb05744cb36bfcbb3 in term 0.
I20260924 04:45:25.093122 20478 leader_election.cc:302] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8a40a6c2f046fa9d789c7f8fc8e3b4, f5aa05d34ac54a1eb05744cb36bfcbb3; no voters:
I20260924 04:45:25.093430 20614 raft_consensus.cc:2781] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:45:25.093515 20614 raft_consensus.cc:482] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:45:25.093583 20614 raft_consensus.cc:3037] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:25.094586 20614 raft_consensus.cc:504] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:25.094934 20614 leader_election.cc:288] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [CANDIDATE]: Term 1 election: Requested vote from peers 9e8a40a6c2f046fa9d789c7f8fc8e3b4 (127.0.98.2:39933), c49a76bd968b4e859b2e857bc16b20ff (127.0.98.1:44555)
I20260924 04:45:25.095482 20412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bdf8c1bbe445a88d7365989ea96be4" candidate_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4"
I20260924 04:45:25.095501 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bdf8c1bbe445a88d7365989ea96be4" candidate_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49a76bd968b4e859b2e857bc16b20ff"
I20260924 04:45:25.095664 20412 raft_consensus.cc:3037] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:25.095835 20281 raft_consensus.cc:3037] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:25.096791 20412 raft_consensus.cc:2454] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5aa05d34ac54a1eb05744cb36bfcbb3 in term 1.
I20260924 04:45:25.096848 20281 raft_consensus.cc:2454] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5aa05d34ac54a1eb05744cb36bfcbb3 in term 1.
I20260924 04:45:25.097137 20478 leader_election.cc:302] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8a40a6c2f046fa9d789c7f8fc8e3b4, f5aa05d34ac54a1eb05744cb36bfcbb3; no voters:
I20260924 04:45:25.097342 20614 raft_consensus.cc:2781] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:25.097584 20614 raft_consensus.cc:686] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 1 LEADER]: Becoming Leader. State: Replica: f5aa05d34ac54a1eb05744cb36bfcbb3, State: Running, Role: LEADER
I20260924 04:45:25.097919 20614 consensus_queue.cc:237] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } }
I20260924 04:45:25.101136 20137 catalog_manager.cc:5495] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5aa05d34ac54a1eb05744cb36bfcbb3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5aa05d34ac54a1eb05744cb36bfcbb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20260924 04:45:25.129202 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.125058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.98.3:37725)
0924 04:45:25.125236 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:25.125243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:25.125260 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:45:25.125350 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:25.125352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:25.125414 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:25.125708 (+ 294us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:25.126044 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.126661 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.127639 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:25.127856 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:25.127971 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:25.128283 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:25.128777 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:25.128923 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:25.128959 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:45:25.129012 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 04:45:25.132787 20459 tablet.cc:2369] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:25.135836 20412 raft_consensus.cc:1261] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 1 FOLLOWER]: Refusing update from remote peer f5aa05d34ac54a1eb05744cb36bfcbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:25.135838 20281 raft_consensus.cc:1261] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 1 FOLLOWER]: Refusing update from remote peer f5aa05d34ac54a1eb05744cb36bfcbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:25.136569 20614 consensus_queue.cc:1035] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8a40a6c2f046fa9d789c7f8fc8e3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:25.136812 20616 consensus_queue.cc:1035] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c49a76bd968b4e859b2e857bc16b20ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:25.144412 20620 mvcc.cc:205] Tried to move back new op lower bound from 7332762112548163584 to 7332762112403546112. Current Snapshot: MvccSnapshot[applied={T|T < 7332762112548163584}]
I20260924 04:45:25.153362 20137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:25.153533 20137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:45:25.155767 20137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2b6210b6eab46e9a054e41617bf8ff6: Sending DeleteTablet for 3 replicas of tablet 78bdf8c1bbe445a88d7365989ea96be4
I20260924 04:45:25.156622 20392 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bdf8c1bbe445a88d7365989ea96be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:25 UTC) from {username='slave'} at 127.0.0.1:36574
I20260924 04:45:25.156716 20261 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bdf8c1bbe445a88d7365989ea96be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:25 UTC) from {username='slave'} at 127.0.0.1:50360
I20260924 04:45:25.157131 20523 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bdf8c1bbe445a88d7365989ea96be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:45:25 UTC) from {username='slave'} at 127.0.0.1:44600
I20260924 04:45:25.157312 20630 tablet_replica.cc:330] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff: stopping tablet replica
I20260924 04:45:25.157680 20630 raft_consensus.cc:2229] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:25.158059 20630 raft_consensus.cc:2258] T 78bdf8c1bbe445a88d7365989ea96be4 P c49a76bd968b4e859b2e857bc16b20ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:25.158336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20199
I20260924 04:45:25.161175 20629 tablet_replica.cc:330] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: stopping tablet replica
I20260924 04:45:25.161423 20631 tablet_replica.cc:330] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: stopping tablet replica
I20260924 04:45:25.161818 20631 raft_consensus.cc:2229] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:45:25.162179 20629 raft_consensus.cc:2229] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:45:25.162678 20631 raft_consensus.cc:2258] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:25.163048 20629 raft_consensus.cc:2258] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:45:25.165117 20629 ts_tablet_manager.cc:1905] T 78bdf8c1bbe445a88d7365989ea96be4 P 9e8a40a6c2f046fa9d789c7f8fc8e3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:45:25.166266 20125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44555 (error 108)
I20260924 04:45:25.166636 20631 ts_tablet_manager.cc:1905] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:45:25.167641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20330
I20260924 04:45:25.167732 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.167416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44555 (local address 127.0.0.1:50370)
0924 04:45:25.167569 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:25.167649 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:45:25.168203 20125 catalog_manager.cc:4584] TS c49a76bd968b4e859b2e857bc16b20ff (127.0.98.1:44555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78bdf8c1bbe445a88d7365989ea96be4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44555: connect: Connection refused (error 111)
I20260924 04:45:25.171499 20631 ts_tablet_manager.cc:1918] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:45:25.171582 20631 log.cc:1196] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Deleting WAL directory at /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224865297037-392-0/minicluster-data/ts-2/wal/wals/78bdf8c1bbe445a88d7365989ea96be4
I20260924 04:45:25.171876 20631 ts_tablet_manager.cc:1939] T 78bdf8c1bbe445a88d7365989ea96be4 P f5aa05d34ac54a1eb05744cb36bfcbb3: Deleting consensus metadata
I20260924 04:45:25.173002 20124 catalog_manager.cc:4841] TS f5aa05d34ac54a1eb05744cb36bfcbb3 (127.0.98.3:37725): tablet 78bdf8c1bbe445a88d7365989ea96be4 (table test-table [id=9545f417da8c49d2a9a624ba832c507f]) successfully deleted
I20260924 04:45:25.177294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20461
I20260924 04:45:25.178750 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:25.178155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39933 (local address 127.0.0.1:36586)
0924 04:45:25.178600 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:25.178662 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20260924 04:45:25.182708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20108
2026-09-24T04:45:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1239 ms)
[----------] 6 tests from EncryptionPolicyTest (11215 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:25 dist-test-slave-npg2 krb5kdc[20635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:25 dist-test-slave-npg2 krb5kdc[20635](info): set up 2 sockets
Sep 24 04:45:25 dist-test-slave-npg2 krb5kdc[20635](info): commencing operation
krb5kdc: starting...
W20260924 04:45:26.231323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:26 dist-test-slave-npg2 krb5kdc[20635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:26.367702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:26.484066 20648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:26.484408 20648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:26.484483 20648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:26.488590 20648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:26.488710 20648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:26.488751 20648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:26.488790 20648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:26.488826 20648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:26.494434 20648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:26.495694 20648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:26.496837 20648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:26.505653 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:26.505708 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:26.505654 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:26.505986 20648 server_base.cc:1034] running on GCE node
I20260924 04:45:26.506565 20648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:26.507855 20648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:26.509060 20648 hybrid_clock.cc:648] HybridClock initialized: now 1790225126509044 us; error 31 us; skew 500 ppm
Sep 24 04:45:26 dist-test-slave-npg2 krb5kdc[20635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:26.512651 20648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:26.513715 20648 webserver.cc:460] Webserver started at http://127.0.0.1:42567/ using document root <none> and password file <none>
I20260924 04:45:26.514271 20648 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:26.514376 20648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:26.514698 20648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:26.516971 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "78bb43bdfd87461d9a34690cbaa79b0b"
format_stamp: "Formatted at 2026-09-24 04:45:26 on dist-test-slave-npg2"
server_key: "45c6d7d53d2dd7777f3113bff2ad960e"
server_key_iv: "956b15236a14375630cd8a64f81b2818"
server_key_version: "encryptionkey@0"
I20260924 04:45:26.517525 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "78bb43bdfd87461d9a34690cbaa79b0b"
format_stamp: "Formatted at 2026-09-24 04:45:26 on dist-test-slave-npg2"
server_key: "45c6d7d53d2dd7777f3113bff2ad960e"
server_key_iv: "956b15236a14375630cd8a64f81b2818"
server_key_version: "encryptionkey@0"
I20260924 04:45:26.521468 20648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 04:45:26.524220 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:26.525681 20648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:26.525799 20648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "78bb43bdfd87461d9a34690cbaa79b0b"
format_stamp: "Formatted at 2026-09-24 04:45:26 on dist-test-slave-npg2"
server_key: "45c6d7d53d2dd7777f3113bff2ad960e"
server_key_iv: "956b15236a14375630cd8a64f81b2818"
server_key_version: "encryptionkey@0"
I20260924 04:45:26.525899 20648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:26.547528 20648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:26.550755 20648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:26.550918 20648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:26.558194 20648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38385
I20260924 04:45:26.558302 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38385 every 8 connection(s)
I20260924 04:45:26.559325 20648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:26.562309 20716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:26.564796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20648
I20260924 04:45:26.564913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:26.565229 392 external_mini_cluster.cc:1428] Setting key 6fecfdff1707fd5d551b3995d887bc24
I20260924 04:45:26.569015 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b: Bootstrap starting.
I20260924 04:45:26.571756 20716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:26.572669 20716 log.cc:824] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:26.574795 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b: No bootstrap required, opened a new log
Sep 24 04:45:26 dist-test-slave-npg2 krb5kdc[20635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:45:26.579676 20716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260924 04:45:26.579974 20716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:26.580027 20716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bb43bdfd87461d9a34690cbaa79b0b, State: Initialized, Role: FOLLOWER
I20260924 04:45:26.580518 20716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260924 04:45:26.580700 20716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:26.580780 20716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:26.580880 20716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:26.582089 20716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260924 04:45:26.582358 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:26.566735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52951 (local address 127.0.0.1:38385)
0924 04:45:26.567225 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:26.567245 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:26.567291 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:45:26.568129 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:26.568164 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:26.568601 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:26.568976 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:26.569716 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:26.571241 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:26.572136 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:26.572840 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:26.572999 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:26.575826 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:26.575832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:26.575849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:26.575881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:26.578776 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:26.579400 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:26.579405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:26.579411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:26.579555 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:26.579925 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:26.579928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:26.579933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:26.580462 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:26.580693 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:26.581067 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 04:45:26.581293 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260924 04:45:26.582451 20716 leader_election.cc:302] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bb43bdfd87461d9a34690cbaa79b0b; no voters:
I20260924 04:45:26.582809 20716 leader_election.cc:288] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:26.582883 20721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:26.583092 20721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [term 1 LEADER]: Becoming Leader. State: Replica: 78bb43bdfd87461d9a34690cbaa79b0b, State: Running, Role: LEADER
I20260924 04:45:26.583441 20721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260924 04:45:26.584192 20716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:26.585383 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } } }
I20260924 04:45:26.585816 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:26.586066 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:26.585706 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bb43bdfd87461d9a34690cbaa79b0b. Latest consensus state: current_term: 1 leader_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bb43bdfd87461d9a34690cbaa79b0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } } }
I20260924 04:45:26.586238 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:26.586304 20734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:26.589838 20734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:26.595886 20734 catalog_manager.cc:1347] Generated new cluster ID: 364075749c064ca6811c0eb3139f5288
I20260924 04:45:26.596017 20734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:26.598059 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:26.588470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48177 (local address 127.0.0.1:38385)
0924 04:45:26.588581 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:26.588586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:26.588847 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:45:26.588970 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:26.588973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:26.589022 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:26.589120 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:26.589925 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:26.590670 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:26.591871 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:26.592207 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:26.592339 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:26.594127 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:26.594132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:26.594138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:26.594171 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:26.595870 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:26.596502 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:26.596506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:26.596508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:26.596596 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:26.597128 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:26.597131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:26.597133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:26.597332 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:26.597471 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:26.597797 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 04:45:26.597888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:45:26.622370 20734 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:26.623368 20734 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:26.637115 20734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bb43bdfd87461d9a34690cbaa79b0b: Generated new TSK 0
I20260924 04:45:26.638125 20734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:26.643339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20648
W20260924 04:45:26.649971 20633 connection.cc:535] client connection to 127.0.0.1:38385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38385): Connection reset by peer (error 104)
2026-09-24T04:45:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:26 dist-test-slave-npg2 krb5kdc[20747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:26 dist-test-slave-npg2 krb5kdc[20747](info): set up 2 sockets
Sep 24 04:45:26 dist-test-slave-npg2 krb5kdc[20747](info): commencing operation
krb5kdc: starting...
W20260924 04:45:27.695580 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:27 dist-test-slave-npg2 krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:27.823395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:27.935953 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:27.936292 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:27.936372 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:27.940454 20760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:27.940567 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:27.940603 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:27.940682 20760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:27.940742 20760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:27.946214 20760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:27.947403 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:27.948447 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:27.957320 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:27.957329 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:27.957384 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:27.957341 20760 server_base.cc:1034] running on GCE node
I20260924 04:45:27.958088 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:27.959213 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:27.960378 20760 hybrid_clock.cc:648] HybridClock initialized: now 1790225127960360 us; error 31 us; skew 500 ppm
Sep 24 04:45:27 dist-test-slave-npg2 krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:27.968263 20760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:27.969264 20760 webserver.cc:460] Webserver started at http://127.0.0.1:36503/ using document root <none> and password file <none>
I20260924 04:45:27.969862 20760 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:27.969925 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:27.970149 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:27.972313 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "3a05db89658c4149a9712f86f86c5d2c"
format_stamp: "Formatted at 2026-09-24 04:45:27 on dist-test-slave-npg2"
server_key: "5f68607c2ee91126093d55653ab46e1b"
server_key_iv: "e74a4c6043702a2b1fe780054004999e"
server_key_version: "encryptionkey@0"
I20260924 04:45:27.972896 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a05db89658c4149a9712f86f86c5d2c"
format_stamp: "Formatted at 2026-09-24 04:45:27 on dist-test-slave-npg2"
server_key: "5f68607c2ee91126093d55653ab46e1b"
server_key_iv: "e74a4c6043702a2b1fe780054004999e"
server_key_version: "encryptionkey@0"
I20260924 04:45:27.976804 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:45:27.979372 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:27.980734 20760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:45:27.980882 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "3a05db89658c4149a9712f86f86c5d2c"
format_stamp: "Formatted at 2026-09-24 04:45:27 on dist-test-slave-npg2"
server_key: "5f68607c2ee91126093d55653ab46e1b"
server_key_iv: "e74a4c6043702a2b1fe780054004999e"
server_key_version: "encryptionkey@0"
I20260924 04:45:27.981000 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:28.007349 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:28.011101 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:28.011407 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:28.018956 20760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36605
I20260924 04:45:28.019037 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36605 every 8 connection(s)
I20260924 04:45:28.020059 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:28.020385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20760
I20260924 04:45:28.020551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:28.020895 392 external_mini_cluster.cc:1428] Setting key 75424a5604c33b0c23177f4f109e4431
I20260924 04:45:28.023432 20828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:28.030475 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c: Bootstrap starting.
I20260924 04:45:28.030838 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:28.022344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33089 (local address 127.0.0.1:36605)
0924 04:45:28.022766 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:28.022784 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:28.022831 (+ 47us) server_negotiation.cc:407] Connection header received
0924 04:45:28.023605 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:28.023625 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:28.023985 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:28.024281 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:28.024606 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:28.025661 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:28.026984 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:28.027660 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:28.027786 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:28.028833 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:28.028838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:28.028852 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:28.029270 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:28.029327 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:28.029351 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:28.029458 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:28.029848 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 04:45:28.030002 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":155,"threads_started":1}
I20260924 04:45:28.033227 20828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:28.034122 20828 log.cc:824] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:28.034305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:28.036505 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c: No bootstrap required, opened a new log
I20260924 04:45:28.041216 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:28.036597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54679 (local address 127.0.0.1:36605)
0924 04:45:28.036728 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:28.036734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:28.037011 (+ 277us) server_negotiation.cc:407] Connection header received
0924 04:45:28.037129 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:28.037132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:28.037172 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:28.037243 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:28.037584 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:28.038122 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:28.039424 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:28.039668 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:28.039757 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:28.040660 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:28.040663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:28.040665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:28.040677 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:28.040703 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:28.040713 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:28.040793 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:28.040971 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:45:28.041046 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:28.041293 20828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a05db89658c4149a9712f86f86c5d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260924 04:45:28.041641 20828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:28.041703 20828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a05db89658c4149a9712f86f86c5d2c, State: Initialized, Role: FOLLOWER
I20260924 04:45:28.042192 20828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a05db89658c4149a9712f86f86c5d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260924 04:45:28.042354 20828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:28.042426 20828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:28.042518 20828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:28.043653 20828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a05db89658c4149a9712f86f86c5d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260924 04:45:28.044059 20828 leader_election.cc:302] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a05db89658c4149a9712f86f86c5d2c; no voters:
I20260924 04:45:28.044440 20828 leader_election.cc:288] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:28.044605 20838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:28.044883 20838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [term 1 LEADER]: Becoming Leader. State: Replica: 3a05db89658c4149a9712f86f86c5d2c, State: Running, Role: LEADER
I20260924 04:45:28.045254 20838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a05db89658c4149a9712f86f86c5d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260924 04:45:28.045791 20828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:28.047014 20840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a05db89658c4149a9712f86f86c5d2c. Latest consensus state: current_term: 1 leader_uuid: "3a05db89658c4149a9712f86f86c5d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a05db89658c4149a9712f86f86c5d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } } }
I20260924 04:45:28.046998 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a05db89658c4149a9712f86f86c5d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a05db89658c4149a9712f86f86c5d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } } }
I20260924 04:45:28.047212 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:28.047640 20840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c [sys.catalog]: This master's current role is: LEADER
W20260924 04:45:28.051388 20853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:28.051472 20853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:28.051663 20849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:28.052421 20849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:28.058316 20849 catalog_manager.cc:1347] Generated new cluster ID: bcdff4bbd8f3419d89fedd090d02b698
I20260924 04:45:28.058390 20849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:28.065970 20849 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:28.067262 20849 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:28.082980 20849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a05db89658c4149a9712f86f86c5d2c: Generated new TSK 0
I20260924 04:45:28.084010 20849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:28.095943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20760
W20260924 04:45:28.101429 20745 connection.cc:535] client connection to 127.0.0.1:36605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36605): Connection reset by peer (error 104)
2026-09-24T04:45:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:28 dist-test-slave-npg2 krb5kdc[20859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:28 dist-test-slave-npg2 krb5kdc[20859](info): set up 2 sockets
Sep 24 04:45:28 dist-test-slave-npg2 krb5kdc[20859](info): commencing operation
krb5kdc: starting...
W20260924 04:45:29.219400 392 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:29 dist-test-slave-npg2 krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:29.346860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:29.459853 20872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:29.460165 20872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:29.460240 20872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:29.464296 20872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:29.464385 20872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:29.464422 20872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:29.464480 20872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:29.464520 20872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:29.469954 20872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:29.471508 20872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:29.473043 20872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:29.482730 20880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:29.482707 20878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:29.482933 20872 server_base.cc:1034] running on GCE node
W20260924 04:45:29.482712 20877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:29.483609 20872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:29.484681 20872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:29.485870 20872 hybrid_clock.cc:648] HybridClock initialized: now 1790225129485850 us; error 33 us; skew 500 ppm
Sep 24 04:45:29 dist-test-slave-npg2 krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:29.488618 20872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:29.489523 20872 webserver.cc:460] Webserver started at http://127.0.0.1:44555/ using document root <none> and password file <none>
I20260924 04:45:29.490088 20872 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:29.490177 20872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:29.490406 20872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:29.492479 20872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "c0625e9048b84d62bacacd3882fe48c2"
format_stamp: "Formatted at 2026-09-24 04:45:29 on dist-test-slave-npg2"
server_key: "d0fa80e3024315a51a82b45407f25c49"
server_key_iv: "680f0de635ae7bee2477c507f6e48a36"
server_key_version: "encryptionkey@0"
I20260924 04:45:29.493084 20872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0625e9048b84d62bacacd3882fe48c2"
format_stamp: "Formatted at 2026-09-24 04:45:29 on dist-test-slave-npg2"
server_key: "d0fa80e3024315a51a82b45407f25c49"
server_key_iv: "680f0de635ae7bee2477c507f6e48a36"
server_key_version: "encryptionkey@0"
I20260924 04:45:29.496881 20872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:29.499348 20887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:29.500665 20872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:29.500813 20872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "c0625e9048b84d62bacacd3882fe48c2"
format_stamp: "Formatted at 2026-09-24 04:45:29 on dist-test-slave-npg2"
server_key: "d0fa80e3024315a51a82b45407f25c49"
server_key_iv: "680f0de635ae7bee2477c507f6e48a36"
server_key_version: "encryptionkey@0"
I20260924 04:45:29.500947 20872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:29.505627 20872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:29.508811 20872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:29.509007 20872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:29.516047 20872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36781
I20260924 04:45:29.516068 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36781 every 8 connection(s)
I20260924 04:45:29.517069 20872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:29.519995 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:29.523377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20872
I20260924 04:45:29.523531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:29.523828 392 external_mini_cluster.cc:1428] Setting key fad0aac928693f8f30a89e7e2dd87663
I20260924 04:45:29.526057 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2: Bootstrap starting.
I20260924 04:45:29.528440 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:29.529224 20940 log.cc:824] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:29.529265 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:29.525360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42145 (local address 127.0.0.1:36781)
0924 04:45:29.525816 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:29.525831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:29.525867 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:45:29.526590 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:29.526609 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:29.526904 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:29.527279 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:29.528074 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:29.528079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:29.528091 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:29.528101 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:29.528152 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:29.528171 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:29.528285 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:29.528406 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:45:29.528561 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":137,"threads_started":1}
I20260924 04:45:29.531369 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2: No bootstrap required, opened a new log
I20260924 04:45:29.532143 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:29.535454 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0625e9048b84d62bacacd3882fe48c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260924 04:45:29.535770 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:29.535836 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0625e9048b84d62bacacd3882fe48c2, State: Initialized, Role: FOLLOWER
I20260924 04:45:29.536262 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:29.534204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57543 (local address 127.0.0.1:36781)
0924 04:45:29.534350 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:29.534354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:29.534511 (+ 157us) server_negotiation.cc:407] Connection header received
0924 04:45:29.534777 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:29.534780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:29.534823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:29.534915 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:29.535792 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:29.535796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:29.535798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:29.535809 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:29.535836 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:29.535853 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:29.535935 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:29.536096 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:45:29.536134 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:45:29.536351 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0625e9048b84d62bacacd3882fe48c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260924 04:45:29.536516 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:29.536592 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:29.536692 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:29.537806 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0625e9048b84d62bacacd3882fe48c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260924 04:45:29.538105 20940 leader_election.cc:302] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0625e9048b84d62bacacd3882fe48c2; no voters:
I20260924 04:45:29.538352 20940 leader_election.cc:288] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:29.538491 20950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:29.538720 20950 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [term 1 LEADER]: Becoming Leader. State: Replica: c0625e9048b84d62bacacd3882fe48c2, State: Running, Role: LEADER
I20260924 04:45:29.539059 20950 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0625e9048b84d62bacacd3882fe48c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } }
I20260924 04:45:29.539716 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:29.540850 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0625e9048b84d62bacacd3882fe48c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0625e9048b84d62bacacd3882fe48c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } } }
I20260924 04:45:29.540977 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:29.541342 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0625e9048b84d62bacacd3882fe48c2. Latest consensus state: current_term: 1 leader_uuid: "c0625e9048b84d62bacacd3882fe48c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0625e9048b84d62bacacd3882fe48c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36781 } } }
I20260924 04:45:29.541538 20959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:29.542109 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:29.544842 20959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:29.550596 20959 catalog_manager.cc:1347] Generated new cluster ID: 9c12b9adfd204fcea69473f19d594c48
I20260924 04:45:29.550679 20959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:29.562131 20959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:29.562491 20959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:29.568233 20959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0625e9048b84d62bacacd3882fe48c2: Generated new TSK 0
I20260924 04:45:29.569041 20959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:29.574137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20872
2026-09-24T04:45:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:29 dist-test-slave-npg2 krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:29 dist-test-slave-npg2 krb5kdc[20971](info): set up 2 sockets
Sep 24 04:45:29 dist-test-slave-npg2 krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260924 04:45:30.635753 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:30 dist-test-slave-npg2 krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:30.762705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:30.874233 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:30.874543 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:30.874588 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:30.878829 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:30.878906 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:30.878926 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:30.878944 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:30.878960 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:30.884650 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:30.885836 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:30.886941 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:30.895915 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:30.895967 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:30.896108 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:30.896155 20984 server_base.cc:1034] running on GCE node
I20260924 04:45:30.896804 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:30.897935 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:30.899204 20984 hybrid_clock.cc:648] HybridClock initialized: now 1790225130899157 us; error 54 us; skew 500 ppm
Sep 24 04:45:30 dist-test-slave-npg2 krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:30.901947 20984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:30.902882 20984 webserver.cc:460] Webserver started at http://127.0.0.1:45925/ using document root <none> and password file <none>
I20260924 04:45:30.903509 20984 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:30.903601 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:30.903836 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:30.905910 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "c54852cd84ef4716ae6bf5484667d86e"
format_stamp: "Formatted at 2026-09-24 04:45:30 on dist-test-slave-npg2"
server_key: "4fee6eb379c59217ed09e1904bc35d52"
server_key_iv: "72b5480ec42622e41746b1dfde8235c2"
server_key_version: "encryptionkey@0"
I20260924 04:45:30.906483 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "c54852cd84ef4716ae6bf5484667d86e"
format_stamp: "Formatted at 2026-09-24 04:45:30 on dist-test-slave-npg2"
server_key: "4fee6eb379c59217ed09e1904bc35d52"
server_key_iv: "72b5480ec42622e41746b1dfde8235c2"
server_key_version: "encryptionkey@0"
I20260924 04:45:30.910404 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:30.912901 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:30.914125 20984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:45:30.914278 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "c54852cd84ef4716ae6bf5484667d86e"
format_stamp: "Formatted at 2026-09-24 04:45:30 on dist-test-slave-npg2"
server_key: "4fee6eb379c59217ed09e1904bc35d52"
server_key_iv: "72b5480ec42622e41746b1dfde8235c2"
server_key_version: "encryptionkey@0"
I20260924 04:45:30.914399 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:30.930369 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:30.934171 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:30.934420 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:30.942214 20984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42395
I20260924 04:45:30.942240 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42395 every 8 connection(s)
I20260924 04:45:30.943329 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:30.946213 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:30.949354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 20984
I20260924 04:45:30.949534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:30.949815 392 external_mini_cluster.cc:1428] Setting key 65c4449953efb83dc723cbba61e97778
I20260924 04:45:30.952596 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e: Bootstrap starting.
I20260924 04:45:30.954840 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:30.955665 21052 log.cc:824] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:30.957613 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e: No bootstrap required, opened a new log
Sep 24 04:45:30 dist-test-slave-npg2 krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:45:30.961951 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54852cd84ef4716ae6bf5484667d86e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42395 } }
I20260924 04:45:30.962255 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:30.962306 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54852cd84ef4716ae6bf5484667d86e, State: Initialized, Role: FOLLOWER
I20260924 04:45:30.962833 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54852cd84ef4716ae6bf5484667d86e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42395 } }
I20260924 04:45:30.963016 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:30.963086 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:30.963214 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:30.964123 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54852cd84ef4716ae6bf5484667d86e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42395 } }
I20260924 04:45:30.964510 21052 leader_election.cc:302] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54852cd84ef4716ae6bf5484667d86e; no voters:
I20260924 04:45:30.964890 21052 leader_election.cc:288] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:30.964987 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:30.965289 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [term 1 LEADER]: Becoming Leader. State: Replica: c54852cd84ef4716ae6bf5484667d86e, State: Running, Role: LEADER
I20260924 04:45:30.965653 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54852cd84ef4716ae6bf5484667d86e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42395 } }
I20260924 04:45:30.966235 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:30.951379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41335 (local address 127.0.0.1:42395)
0924 04:45:30.951869 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:30.951884 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:30.951921 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:45:30.952546 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:30.952564 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:30.952984 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:30.953336 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:30.954251 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:30.955405 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:30.956362 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:30.956902 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:30.957239 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:30.960131 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:30.960137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:30.960154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:30.960185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:30.962792 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:30.963353 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:30.963364 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:30.963368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:30.963487 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:30.963789 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:30.963793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:30.963795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:30.964210 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:30.964411 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:30.964783 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 04:45:30.965138 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":130,"threads_started":1}
I20260924 04:45:30.966233 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:30.966962 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54852cd84ef4716ae6bf5484667d86e. Latest consensus state: current_term: 1 leader_uuid: "c54852cd84ef4716ae6bf5484667d86e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54852cd84ef4716ae6bf5484667d86e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42395 } } }
I20260924 04:45:30.967099 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:30.967463 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54852cd84ef4716ae6bf5484667d86e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54852cd84ef4716ae6bf5484667d86e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42395 } } }
I20260924 04:45:30.967556 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:30.967854 21061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:30.969820 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:30.971097 21061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:30.977785 21061 catalog_manager.cc:1347] Generated new cluster ID: 678cb25880d84ffbac3b7368391bfb90
I20260924 04:45:30.977859 21061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:30.982225 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:30.972356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42957 (local address 127.0.0.1:42395)
0924 04:45:30.972479 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:30.972483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:30.973736 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 04:45:30.973856 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:30.973860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:30.973907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:30.973995 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:30.974863 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:30.975585 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:30.976540 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:30.976802 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:30.976920 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:30.978563 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:30.978567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:30.978570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:30.978597 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:30.980469 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:30.980935 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:30.980939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:30.980942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:30.981036 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:30.981326 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:30.981330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:30.981331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:30.981558 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:30.981693 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:30.981915 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 04:45:30.982044 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:45:30.986680 21061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:30.987686 21061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:31.001938 21061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54852cd84ef4716ae6bf5484667d86e: Generated new TSK 0
I20260924 04:45:31.003082 21061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:31.012813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 20984
2026-09-24T04:45:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:31 dist-test-slave-npg2 krb5kdc[21083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:31 dist-test-slave-npg2 krb5kdc[21083](info): set up 2 sockets
Sep 24 04:45:31 dist-test-slave-npg2 krb5kdc[21083](info): commencing operation
krb5kdc: starting...
W20260924 04:45:32.071521 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:32 dist-test-slave-npg2 krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:32.199890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:32.314229 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:32.314507 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:32.314549 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:32.318909 21096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:32.318987 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:32.319006 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:32.319024 21096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:32.319041 21096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:32.324606 21096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:32.325757 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:32.326820 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:32.335736 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:32.335804 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:32.335788 21096 server_base.cc:1034] running on GCE node
W20260924 04:45:32.335747 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:32.336702 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:32.337884 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:32.339093 21096 hybrid_clock.cc:648] HybridClock initialized: now 1790225132339049 us; error 50 us; skew 500 ppm
Sep 24 04:45:32 dist-test-slave-npg2 krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:32.342562 21096 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:32.343683 21096 webserver.cc:460] Webserver started at http://127.0.0.1:35701/ using document root <none> and password file <none>
I20260924 04:45:32.344308 21096 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:32.344373 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:32.344590 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:32.346724 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "90fd25a8f8ed47a1a6468af638b1acc4"
format_stamp: "Formatted at 2026-09-24 04:45:32 on dist-test-slave-npg2"
server_key: "4b2524e06acd932feded30303def1f5c"
server_key_iv: "d4cae4ffce1ef91428c1c099178959e0"
server_key_version: "encryptionkey@0"
I20260924 04:45:32.347352 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "90fd25a8f8ed47a1a6468af638b1acc4"
format_stamp: "Formatted at 2026-09-24 04:45:32 on dist-test-slave-npg2"
server_key: "4b2524e06acd932feded30303def1f5c"
server_key_iv: "d4cae4ffce1ef91428c1c099178959e0"
server_key_version: "encryptionkey@0"
I20260924 04:45:32.352891 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 04:45:32.355779 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:32.357075 21096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:45:32.357231 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "90fd25a8f8ed47a1a6468af638b1acc4"
format_stamp: "Formatted at 2026-09-24 04:45:32 on dist-test-slave-npg2"
server_key: "4b2524e06acd932feded30303def1f5c"
server_key_iv: "d4cae4ffce1ef91428c1c099178959e0"
server_key_version: "encryptionkey@0"
I20260924 04:45:32.357348 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:32.371030 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:32.374423 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:32.374629 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:32.381876 21096 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40239
I20260924 04:45:32.381899 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40239 every 8 connection(s)
I20260924 04:45:32.382902 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:32.385730 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:32.386264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21096
I20260924 04:45:32.386415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:32.386703 392 external_mini_cluster.cc:1428] Setting key 610f0eca40e7b905c7c71a1a17c53576
I20260924 04:45:32.393105 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4: Bootstrap starting.
I20260924 04:45:32.395831 21164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:32.396394 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:32.388153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40477 (local address 127.0.0.1:40239)
0924 04:45:32.388588 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:32.388603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:32.388651 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:45:32.389353 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:32.389373 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:32.389754 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:32.390048 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:32.390494 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:32.391673 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:32.392885 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:32.393486 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:32.393629 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:32.394505 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:32.394509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:32.394522 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:32.394979 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:32.395034 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:32.395058 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:32.395221 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:32.395342 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:45:32.395565 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260924 04:45:32.396648 21164 log.cc:824] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:32.399251 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:32.399462 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4: No bootstrap required, opened a new log
I20260924 04:45:32.403510 21164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40239 } }
I20260924 04:45:32.403779 21164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:32.403858 21164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90fd25a8f8ed47a1a6468af638b1acc4, State: Initialized, Role: FOLLOWER
I20260924 04:45:32.404337 21164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40239 } }
I20260924 04:45:32.404490 21164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:32.404578 21164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:32.404688 21164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:32.405412 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:32.401347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34557 (local address 127.0.0.1:40239)
0924 04:45:32.401474 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:32.401480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:32.401718 (+ 238us) server_negotiation.cc:407] Connection header received
0924 04:45:32.401778 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:32.401780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:32.401827 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:32.401909 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:32.402196 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:32.402717 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:32.403634 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:32.403962 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:32.404064 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:32.404905 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:32.404908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:32.404910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:32.404922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:32.404952 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:32.404963 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:32.405022 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:32.405192 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:45:32.405240 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:45:32.405931 21164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40239 } }
I20260924 04:45:32.406306 21164 leader_election.cc:302] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90fd25a8f8ed47a1a6468af638b1acc4; no voters:
I20260924 04:45:32.406628 21164 leader_election.cc:288] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:32.406749 21174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:32.406958 21174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [term 1 LEADER]: Becoming Leader. State: Replica: 90fd25a8f8ed47a1a6468af638b1acc4, State: Running, Role: LEADER
I20260924 04:45:32.407323 21174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40239 } }
I20260924 04:45:32.407990 21164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:32.409188 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90fd25a8f8ed47a1a6468af638b1acc4. Latest consensus state: current_term: 1 leader_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40239 } } }
I20260924 04:45:32.409171 21175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fd25a8f8ed47a1a6468af638b1acc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40239 } } }
I20260924 04:45:32.409334 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:32.409338 21175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:32.410426 21183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:32.413163 21183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:32.419313 21183 catalog_manager.cc:1347] Generated new cluster ID: f39918b79e13466cb80c6fdc3ca040c0
I20260924 04:45:32.419394 21183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:32.442026 21183 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:32.443485 21183 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:32.456034 21183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90fd25a8f8ed47a1a6468af638b1acc4: Generated new TSK 0
I20260924 04:45:32.456954 21183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:32.461292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21096
2026-09-24T04:45:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:32 dist-test-slave-npg2 krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:32 dist-test-slave-npg2 krb5kdc[21195](info): set up 2 sockets
Sep 24 04:45:32 dist-test-slave-npg2 krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260924 04:45:33.507889 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:33 dist-test-slave-npg2 krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:33.642776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:33.753726 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:33.754070 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:33.754155 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:33.758266 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:33.758378 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:33.758419 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:33.758473 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:33.758507 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:33.763978 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:33.765125 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:33.766227 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:33.775252 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:33.775312 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:33.775341 21208 server_base.cc:1034] running on GCE node
W20260924 04:45:33.775550 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:33.776103 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:33.777320 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:33.778541 21208 hybrid_clock.cc:648] HybridClock initialized: now 1790225133778511 us; error 43 us; skew 500 ppm
Sep 24 04:45:33 dist-test-slave-npg2 krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:33.786913 21208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:33.788072 21208 webserver.cc:460] Webserver started at http://127.0.0.1:44531/ using document root <none> and password file <none>
I20260924 04:45:33.788690 21208 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:33.788789 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:33.789044 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:33.791469 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "2eee62345e3f47c9a51121c3d25abb62"
format_stamp: "Formatted at 2026-09-24 04:45:33 on dist-test-slave-npg2"
server_key: "b3cbb75e26ca479c4196152e32b1b9d7"
server_key_iv: "c374056bcbeb4cfafeb3f1f346dc0a1e"
server_key_version: "encryptionkey@0"
I20260924 04:45:33.792230 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "2eee62345e3f47c9a51121c3d25abb62"
format_stamp: "Formatted at 2026-09-24 04:45:33 on dist-test-slave-npg2"
server_key: "b3cbb75e26ca479c4196152e32b1b9d7"
server_key_iv: "c374056bcbeb4cfafeb3f1f346dc0a1e"
server_key_version: "encryptionkey@0"
I20260924 04:45:33.796159 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:45:33.798888 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:33.800307 21208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:45:33.800468 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "2eee62345e3f47c9a51121c3d25abb62"
format_stamp: "Formatted at 2026-09-24 04:45:33 on dist-test-slave-npg2"
server_key: "b3cbb75e26ca479c4196152e32b1b9d7"
server_key_iv: "c374056bcbeb4cfafeb3f1f346dc0a1e"
server_key_version: "encryptionkey@0"
I20260924 04:45:33.800595 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:33.805358 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:33.808780 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:33.808992 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:33.816337 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43339
I20260924 04:45:33.816357 21275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43339 every 8 connection(s)
I20260924 04:45:33.817292 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:33.819279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21208
I20260924 04:45:33.819442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:33.819782 392 external_mini_cluster.cc:1428] Setting key 99e19d740ce06db66bbc3f04189b93fd
I20260924 04:45:33.820225 21276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:33.825556 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:33.821217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43885 (local address 127.0.0.1:43339)
0924 04:45:33.821636 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:33.821655 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:33.821698 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:45:33.822534 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:33.822557 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:33.822918 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:33.823381 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:33.824110 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:33.824116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:33.824131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:33.824144 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:33.824208 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:33.824250 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:33.824397 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:33.824465 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 04:45:33.824609 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260924 04:45:33.827663 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62: Bootstrap starting.
I20260924 04:45:33.828657 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:33.830092 21276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:33.830919 21276 log.cc:824] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:33.832628 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:33.830758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48741 (local address 127.0.0.1:43339)
0924 04:45:33.830886 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:33.830891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:33.831061 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:45:33.831235 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:33.831238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:33.831273 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:33.831361 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:33.832186 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:33.832190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:33.832192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:33.832204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:33.832277 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:33.832291 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:33.832379 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:33.832468 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:45:33.832503 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:33.833231 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62: No bootstrap required, opened a new log
I20260924 04:45:33.836858 21276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee62345e3f47c9a51121c3d25abb62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260924 04:45:33.837117 21276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:33.837201 21276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eee62345e3f47c9a51121c3d25abb62, State: Initialized, Role: FOLLOWER
I20260924 04:45:33.837657 21276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee62345e3f47c9a51121c3d25abb62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260924 04:45:33.837827 21276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:33.837916 21276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:33.838037 21276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:33.838994 21276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee62345e3f47c9a51121c3d25abb62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260924 04:45:33.839367 21276 leader_election.cc:302] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eee62345e3f47c9a51121c3d25abb62; no voters:
I20260924 04:45:33.839735 21276 leader_election.cc:288] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:33.839972 21286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:33.840240 21286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [term 1 LEADER]: Becoming Leader. State: Replica: 2eee62345e3f47c9a51121c3d25abb62, State: Running, Role: LEADER
I20260924 04:45:33.840575 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee62345e3f47c9a51121c3d25abb62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260924 04:45:33.840958 21276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:33.842226 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eee62345e3f47c9a51121c3d25abb62. Latest consensus state: current_term: 1 leader_uuid: "2eee62345e3f47c9a51121c3d25abb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee62345e3f47c9a51121c3d25abb62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } } }
I20260924 04:45:33.842283 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eee62345e3f47c9a51121c3d25abb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee62345e3f47c9a51121c3d25abb62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } } }
I20260924 04:45:33.842396 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:33.842347 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:33.842772 21295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:33.845950 21295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:33.851459 21295 catalog_manager.cc:1347] Generated new cluster ID: 512b2927a139480e918facacf85e30d8
I20260924 04:45:33.851558 21295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:33.861127 21295 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:33.861534 21295 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:33.871574 21295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eee62345e3f47c9a51121c3d25abb62: Generated new TSK 0
I20260924 04:45:33.872538 21295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:33.882596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21208
2026-09-24T04:45:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:33 dist-test-slave-npg2 krb5kdc[21307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:33 dist-test-slave-npg2 krb5kdc[21307](info): set up 2 sockets
Sep 24 04:45:33 dist-test-slave-npg2 krb5kdc[21307](info): commencing operation
krb5kdc: starting...
W20260924 04:45:34.987478 392 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:35 dist-test-slave-npg2 krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:35.114820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:35.228184 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:35.228530 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:35.228621 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:35.232788 21320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:35.232897 21320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:35.232937 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:35.232975 21320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:35.233011 21320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:35.238827 21320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:35.240011 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:35.241084 21320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:35.250429 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:35.250526 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:35.250690 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:35.250841 21320 server_base.cc:1034] running on GCE node
I20260924 04:45:35.251516 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:35.252792 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:35.254007 21320 hybrid_clock.cc:648] HybridClock initialized: now 1790225135253994 us; error 31 us; skew 500 ppm
Sep 24 04:45:35 dist-test-slave-npg2 krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:35.256939 21320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:35.257882 21320 webserver.cc:460] Webserver started at http://127.0.0.1:37603/ using document root <none> and password file <none>
I20260924 04:45:35.258478 21320 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:35.258574 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:35.258807 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:35.261037 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "da6495658d8249b3ac0c96b27de1dd2b"
format_stamp: "Formatted at 2026-09-24 04:45:35 on dist-test-slave-npg2"
server_key: "e7dac16272e81547112037545fced911"
server_key_iv: "0fd041b18d84d44d6c0b8d49fb869b59"
server_key_version: "encryptionkey@0"
I20260924 04:45:35.261628 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "da6495658d8249b3ac0c96b27de1dd2b"
format_stamp: "Formatted at 2026-09-24 04:45:35 on dist-test-slave-npg2"
server_key: "e7dac16272e81547112037545fced911"
server_key_iv: "0fd041b18d84d44d6c0b8d49fb869b59"
server_key_version: "encryptionkey@0"
I20260924 04:45:35.265515 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:35.268096 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:35.269466 21320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:35.269637 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "da6495658d8249b3ac0c96b27de1dd2b"
format_stamp: "Formatted at 2026-09-24 04:45:35 on dist-test-slave-npg2"
server_key: "e7dac16272e81547112037545fced911"
server_key_iv: "0fd041b18d84d44d6c0b8d49fb869b59"
server_key_version: "encryptionkey@0"
I20260924 04:45:35.269766 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:35.288093 21320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:35.291499 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:35.291728 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:35.299649 21320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40273
I20260924 04:45:35.299664 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40273 every 8 connection(s)
I20260924 04:45:35.300668 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:35.301412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21320
I20260924 04:45:35.301558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:35.301855 392 external_mini_cluster.cc:1428] Setting key cdf0eb4858c23f6d3b0a1d7e75e4f33b
I20260924 04:45:35.304955 21388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 04:45:35 dist-test-slave-npg2 krb5kdc[21307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:45:35.312132 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b: Bootstrap starting.
I20260924 04:45:35.314599 21388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:35.315454 21388 log.cc:824] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:35.317049 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:35.303447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:55707 (local address 127.0.0.1:40273)
0924 04:45:35.303975 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:35.304002 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:35.304042 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:45:35.304679 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:35.304696 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:35.305131 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:35.305457 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:35.306380 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:35.307506 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:35.308689 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:35.309172 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:35.309319 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:35.312225 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:35.312230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:35.312243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:35.312302 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:35.314032 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:35.314574 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:35.314579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:35.314583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:35.314755 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:35.315109 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:35.315113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:35.315114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:35.315547 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:35.315792 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:35.316099 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:45:35.316324 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":199,"threads_started":1}
I20260924 04:45:35.317615 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b: No bootstrap required, opened a new log
I20260924 04:45:35.320238 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:35.322373 21388 raft_consensus.cc:348] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6495658d8249b3ac0c96b27de1dd2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260924 04:45:35.322664 21388 raft_consensus.cc:374] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:35.322757 21388 raft_consensus.cc:729] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6495658d8249b3ac0c96b27de1dd2b, State: Initialized, Role: FOLLOWER
I20260924 04:45:35.323289 21388 consensus_queue.cc:260] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6495658d8249b3ac0c96b27de1dd2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260924 04:45:35.323421 21388 raft_consensus.cc:388] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:35.323505 21388 raft_consensus.cc:482] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:35.323633 21388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:35.324599 21388 raft_consensus.cc:504] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6495658d8249b3ac0c96b27de1dd2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260924 04:45:35.324954 21388 leader_election.cc:302] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da6495658d8249b3ac0c96b27de1dd2b; no voters:
I20260924 04:45:35.325300 21388 leader_election.cc:288] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:35.325373 21398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:35.325572 21398 raft_consensus.cc:686] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [term 1 LEADER]: Becoming Leader. State: Replica: da6495658d8249b3ac0c96b27de1dd2b, State: Running, Role: LEADER
I20260924 04:45:35.325924 21398 consensus_queue.cc:237] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6495658d8249b3ac0c96b27de1dd2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260924 04:45:35.326660 21388 sys_catalog.cc:564] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:35.327701 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader da6495658d8249b3ac0c96b27de1dd2b. Latest consensus state: current_term: 1 leader_uuid: "da6495658d8249b3ac0c96b27de1dd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6495658d8249b3ac0c96b27de1dd2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260924 04:45:35.327896 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:35.328358 21407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:35.328130 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da6495658d8249b3ac0c96b27de1dd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6495658d8249b3ac0c96b27de1dd2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260924 04:45:35.328610 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:35.331681 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:35.322459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:41701 (local address 127.0.0.1:40273)
0924 04:45:35.322593 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:35.322599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:35.322860 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:45:35.323018 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:35.323026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:35.323068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:35.323175 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:35.323963 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:35.324550 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:35.325572 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:35.325890 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:35.326047 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:35.327730 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:35.327736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:35.327739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:35.327767 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:35.329410 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:35.329981 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:35.329984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:35.329986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:35.330094 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:35.330432 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:35.330437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:35.330441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:35.330648 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:35.330787 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:35.331428 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 04:45:35.331520 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:35.332304 21407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:35.338518 21407 catalog_manager.cc:1347] Generated new cluster ID: 5b080924d1264d61a4b9324609c77404
I20260924 04:45:35.338594 21407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:35.352697 21407 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:35.353981 21407 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:35.362758 21407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da6495658d8249b3ac0c96b27de1dd2b: Generated new TSK 0
I20260924 04:45:35.363559 21407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:35.371621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21320
W20260924 04:45:35.377429 21305 connection.cc:535] client connection to 127.0.0.1:40273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40273): Connection reset by peer (error 104)
2026-09-24T04:45:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:35 dist-test-slave-npg2 krb5kdc[21419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:35 dist-test-slave-npg2 krb5kdc[21419](info): set up 2 sockets
Sep 24 04:45:35 dist-test-slave-npg2 krb5kdc[21419](info): commencing operation
krb5kdc: starting...
W20260924 04:45:36.411473 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:36 dist-test-slave-npg2 krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:36.538682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:36.654465 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:36.654846 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:36.654959 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:36.659106 21432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:36.659263 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:36.659312 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:36.659348 21432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:36.659384 21432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:36.664935 21432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:36.666172 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:36.667282 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:45:36.675906 21432 server_base.cc:1034] running on GCE node
W20260924 04:45:36.675843 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:36.675848 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:36.675849 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:36.676641 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:36.677762 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:36.678949 21432 hybrid_clock.cc:648] HybridClock initialized: now 1790225136678924 us; error 37 us; skew 500 ppm
Sep 24 04:45:36 dist-test-slave-npg2 krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:36.682036 21432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:36.683069 21432 webserver.cc:460] Webserver started at http://127.0.0.1:40509/ using document root <none> and password file <none>
I20260924 04:45:36.683748 21432 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:36.683851 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:36.684115 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:36.686323 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "3e76680b7ffa4e5ab974b4daea17bdf8"
format_stamp: "Formatted at 2026-09-24 04:45:36 on dist-test-slave-npg2"
server_key: "c242fd7750802d69752a31f31f0d0aea"
server_key_iv: "4176e5b66030bc9af4926882e13f24bf"
server_key_version: "encryptionkey@0"
I20260924 04:45:36.686955 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e76680b7ffa4e5ab974b4daea17bdf8"
format_stamp: "Formatted at 2026-09-24 04:45:36 on dist-test-slave-npg2"
server_key: "c242fd7750802d69752a31f31f0d0aea"
server_key_iv: "4176e5b66030bc9af4926882e13f24bf"
server_key_version: "encryptionkey@0"
I20260924 04:45:36.691045 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 04:45:36.693729 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:36.695174 21432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:45:36.695278 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "3e76680b7ffa4e5ab974b4daea17bdf8"
format_stamp: "Formatted at 2026-09-24 04:45:36 on dist-test-slave-npg2"
server_key: "c242fd7750802d69752a31f31f0d0aea"
server_key_iv: "4176e5b66030bc9af4926882e13f24bf"
server_key_version: "encryptionkey@0"
I20260924 04:45:36.695374 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:36.714579 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:36.717975 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:36.718129 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:36.725401 21432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42233
I20260924 04:45:36.725442 21499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42233 every 8 connection(s)
I20260924 04:45:36.726394 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:36.729270 21500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:36.734712 21500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8: Bootstrap starting.
I20260924 04:45:36.735598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21432
I20260924 04:45:36.735737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:36.736014 392 external_mini_cluster.cc:1428] Setting key e868d75d7aaa07435f001bd9352720c0
I20260924 04:45:36.737540 21500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:36.738286 21500 log.cc:824] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:36.740161 21500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8: No bootstrap required, opened a new log
I20260924 04:45:36.743693 21500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20260924 04:45:36.743937 21500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:36.744025 21500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e76680b7ffa4e5ab974b4daea17bdf8, State: Initialized, Role: FOLLOWER
I20260924 04:45:36.744475 21500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20260924 04:45:36.744632 21500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:36.744717 21500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:36.744833 21500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:36.745044 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:36.737142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:40339 (local address 127.0.0.1:42233)
0924 04:45:36.737576 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:36.737594 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:36.737629 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:45:36.738339 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:36.738358 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:36.738753 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:36.739089 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:36.739527 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:36.740523 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:36.741419 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:36.741911 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:36.742089 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:36.743040 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:36.743045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:36.743059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:36.743524 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:36.743577 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:36.743601 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:36.743746 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:36.743908 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:45:36.744048 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":149,"threads_started":1}
I20260924 04:45:36.745985 21500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20260924 04:45:36.746344 21500 leader_election.cc:302] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e76680b7ffa4e5ab974b4daea17bdf8; no voters:
I20260924 04:45:36.746709 21500 leader_election.cc:288] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:36.746778 21505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:36.746984 21505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e76680b7ffa4e5ab974b4daea17bdf8, State: Running, Role: LEADER
I20260924 04:45:36.747366 21505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20260924 04:45:36.748050 21500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:36.748517 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:36.749147 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } } }
I20260924 04:45:36.749281 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:36.750640 21506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e76680b7ffa4e5ab974b4daea17bdf8. Latest consensus state: current_term: 1 leader_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e76680b7ffa4e5ab974b4daea17bdf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } } }
I20260924 04:45:36.751387 21506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8 [sys.catalog]: This master's current role is: LEADER
W20260924 04:45:36.753638 21525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:36.753736 21525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:36.754179 21514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:36.755246 21514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:36.755640 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:36.751209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:58343 (local address 127.0.0.1:42233)
0924 04:45:36.751323 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:36.751327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:36.751628 (+ 301us) server_negotiation.cc:407] Connection header received
0924 04:45:36.751812 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:36.751815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:36.751868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:36.751950 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:36.752236 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:36.752763 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:36.753626 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:36.753949 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:36.754062 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:36.754995 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:36.754999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:36.755002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:36.755016 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:36.755042 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:36.755067 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:36.755176 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:36.755461 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 04:45:36.755500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:45:36.761642 21514 catalog_manager.cc:1347] Generated new cluster ID: e00b5899b5fe49aeaecf2c84672c19ef
I20260924 04:45:36.761776 21514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:36.792212 21514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:36.793239 21514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:36.800325 21514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e76680b7ffa4e5ab974b4daea17bdf8: Generated new TSK 0
I20260924 04:45:36.801045 21514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:36.813143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21432
W20260924 04:45:36.818104 21417 connection.cc:535] client connection to 127.0.0.1:42233 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42233): Connection reset by peer (error 104)
2026-09-24T04:45:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:36 dist-test-slave-npg2 krb5kdc[21531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:36 dist-test-slave-npg2 krb5kdc[21531](info): set up 2 sockets
Sep 24 04:45:36 dist-test-slave-npg2 krb5kdc[21531](info): commencing operation
krb5kdc: starting...
W20260924 04:45:37.859586 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:37 dist-test-slave-npg2 krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:37.986788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:38.102347 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:38.102617 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:38.102659 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:38.106664 21544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:38.106736 21544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:38.106757 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:38.106774 21544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:38.106789 21544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:38.112461 21544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:38.113515 21544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:38.114528 21544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:38.123535 21552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:38.123543 21550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:38.123553 21549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:38.123593 21544 server_base.cc:1034] running on GCE node
I20260924 04:45:38.124480 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:38.125627 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:38.126839 21544 hybrid_clock.cc:648] HybridClock initialized: now 1790225138126809 us; error 42 us; skew 500 ppm
Sep 24 04:45:38 dist-test-slave-npg2 krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:38.129779 21544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:38.130811 21544 webserver.cc:460] Webserver started at http://127.0.0.1:34153/ using document root <none> and password file <none>
I20260924 04:45:38.131466 21544 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:38.131562 21544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:38.131863 21544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:38.134027 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "24a3b89a5f874b548bacfdc5d05d4af1"
format_stamp: "Formatted at 2026-09-24 04:45:38 on dist-test-slave-npg2"
server_key: "199b4e2bb9228e0094a9bd08ccd9669f"
server_key_iv: "9c373659c74259f4bb8c4d6c53d6f6bf"
server_key_version: "encryptionkey@0"
I20260924 04:45:38.134645 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "24a3b89a5f874b548bacfdc5d05d4af1"
format_stamp: "Formatted at 2026-09-24 04:45:38 on dist-test-slave-npg2"
server_key: "199b4e2bb9228e0094a9bd08ccd9669f"
server_key_iv: "9c373659c74259f4bb8c4d6c53d6f6bf"
server_key_version: "encryptionkey@0"
I20260924 04:45:38.138612 21544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:45:38.141232 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:38.142556 21544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:45:38.142714 21544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "24a3b89a5f874b548bacfdc5d05d4af1"
format_stamp: "Formatted at 2026-09-24 04:45:38 on dist-test-slave-npg2"
server_key: "199b4e2bb9228e0094a9bd08ccd9669f"
server_key_iv: "9c373659c74259f4bb8c4d6c53d6f6bf"
server_key_version: "encryptionkey@0"
I20260924 04:45:38.142836 21544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:38.147670 21544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:38.150615 21544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:38.150820 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:38.158389 21544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33803
I20260924 04:45:38.158411 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33803 every 8 connection(s)
I20260924 04:45:38.159387 21544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:38.162194 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:38.163422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21544
I20260924 04:45:38.163568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:38.163863 392 external_mini_cluster.cc:1428] Setting key 33b164019308a42abe839722e6f34cb5
I20260924 04:45:38.168756 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1: Bootstrap starting.
I20260924 04:45:38.169312 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:38.165344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:41103 (local address 127.0.0.1:33803)
0924 04:45:38.165858 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:38.165873 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:38.165907 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:45:38.166540 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:38.166557 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:38.166852 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:38.167213 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:38.168038 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:38.168042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:38.168054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:38.168064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:38.168115 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:38.168135 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:38.168278 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:38.168349 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:45:38.168514 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":138,"threads_started":1}
I20260924 04:45:38.171094 21612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:38.171922 21612 log.cc:824] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:38.172169 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:38.173956 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1: No bootstrap required, opened a new log
I20260924 04:45:38.176070 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:38.174333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:33881 (local address 127.0.0.1:33803)
0924 04:45:38.174459 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:38.174463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:38.174698 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:45:38.174746 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:38.174748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:38.174778 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:38.174837 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:38.175680 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:38.175683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:38.175686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:38.175696 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:38.175719 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:38.175729 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:38.175800 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:38.175911 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 04:45:38.175942 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:45:38.178244 21612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33803 } }
I20260924 04:45:38.178508 21612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:38.178567 21612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a3b89a5f874b548bacfdc5d05d4af1, State: Initialized, Role: FOLLOWER
I20260924 04:45:38.179031 21612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33803 } }
I20260924 04:45:38.179214 21612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:38.179304 21612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:38.179405 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:38.180344 21612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33803 } }
I20260924 04:45:38.180714 21612 leader_election.cc:302] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a3b89a5f874b548bacfdc5d05d4af1; no voters:
I20260924 04:45:38.181047 21612 leader_election.cc:288] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:38.181209 21622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:38.181488 21622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [term 1 LEADER]: Becoming Leader. State: Replica: 24a3b89a5f874b548bacfdc5d05d4af1, State: Running, Role: LEADER
I20260924 04:45:38.181836 21622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33803 } }
I20260924 04:45:38.182307 21612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:38.183531 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a3b89a5f874b548bacfdc5d05d4af1. Latest consensus state: current_term: 1 leader_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33803 } } }
I20260924 04:45:38.183669 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:38.184027 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a3b89a5f874b548bacfdc5d05d4af1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33803 } } }
I20260924 04:45:38.184160 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:38.184101 21631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:38.187613 21631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:38.193061 21631 catalog_manager.cc:1347] Generated new cluster ID: 36ceb16c999d47b4ae0f82f25f5e915c
I20260924 04:45:38.193127 21631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:38.198844 21631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:38.199391 21631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:38.211410 21631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a3b89a5f874b548bacfdc5d05d4af1: Generated new TSK 0
I20260924 04:45:38.212181 21631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:38.220680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21544
2026-09-24T04:45:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:38 dist-test-slave-npg2 krb5kdc[21643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:38 dist-test-slave-npg2 krb5kdc[21643](info): set up 2 sockets
Sep 24 04:45:38 dist-test-slave-npg2 krb5kdc[21643](info): commencing operation
krb5kdc: starting...
W20260924 04:45:39.267445 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:39 dist-test-slave-npg2 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:39.398329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:39.508636 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:39.508955 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:39.509029 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:39.513774 21656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:39.513852 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:39.513881 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:39.513907 21656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:39.513942 21656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:39.520248 21656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:39.521400 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:39.522455 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:39.531564 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:39.531682 21656 server_base.cc:1034] running on GCE node
W20260924 04:45:39.531574 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:39.531564 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:39.532673 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:39.533917 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:39.535107 21656 hybrid_clock.cc:648] HybridClock initialized: now 1790225139535099 us; error 37 us; skew 500 ppm
Sep 24 04:45:39 dist-test-slave-npg2 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:39.544647 21656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:39.545696 21656 webserver.cc:460] Webserver started at http://127.0.0.1:43151/ using document root <none> and password file <none>
I20260924 04:45:39.546320 21656 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:39.546433 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:39.546698 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:39.548919 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "f802e0eabf9d4b82a95d24237795bd06"
format_stamp: "Formatted at 2026-09-24 04:45:39 on dist-test-slave-npg2"
server_key: "4bf5723fcc1c7dd22cde15ddaa6b3cce"
server_key_iv: "5cd9358a2c00bd73e9122be9af152a44"
server_key_version: "encryptionkey@0"
I20260924 04:45:39.549494 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "f802e0eabf9d4b82a95d24237795bd06"
format_stamp: "Formatted at 2026-09-24 04:45:39 on dist-test-slave-npg2"
server_key: "4bf5723fcc1c7dd22cde15ddaa6b3cce"
server_key_iv: "5cd9358a2c00bd73e9122be9af152a44"
server_key_version: "encryptionkey@0"
I20260924 04:45:39.553570 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 04:45:39.556413 21671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:39.557735 21656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:45:39.557914 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "f802e0eabf9d4b82a95d24237795bd06"
format_stamp: "Formatted at 2026-09-24 04:45:39 on dist-test-slave-npg2"
server_key: "4bf5723fcc1c7dd22cde15ddaa6b3cce"
server_key_iv: "5cd9358a2c00bd73e9122be9af152a44"
server_key_version: "encryptionkey@0"
I20260924 04:45:39.558043 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:39.569689 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:39.572942 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:39.573156 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:39.580924 21656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36633
I20260924 04:45:39.580941 21723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36633 every 8 connection(s)
I20260924 04:45:39.582044 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:39.584762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21656
I20260924 04:45:39.584926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:39.585189 21724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:39.585207 392 external_mini_cluster.cc:1428] Setting key 61df5815e63657f806f43ff7804116e4
I20260924 04:45:39.592387 21724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06: Bootstrap starting.
Sep 24 04:45:39 dist-test-slave-npg2 krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:45:39.595353 21724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:39.596289 21724 log.cc:824] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:39.598577 21724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06: No bootstrap required, opened a new log
I20260924 04:45:39.600683 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:39.586652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:59129 (local address 127.0.0.1:36633)
0924 04:45:39.587084 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:39.587099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:39.587177 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:45:39.587934 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:39.587953 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:39.588300 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:39.588592 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:39.589573 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:39.590609 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:39.591833 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:39.592309 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:39.592428 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:39.595581 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:39.595587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:39.595607 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:39.595652 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:39.597508 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:39.598026 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:39.598031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:39.598037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:39.598167 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:39.598485 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:39.598487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:39.598489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:39.598992 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:39.599224 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:39.599508 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 04:45:39.599738 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":156,"threads_started":1}
I20260924 04:45:39.602934 21724 raft_consensus.cc:348] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f802e0eabf9d4b82a95d24237795bd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260924 04:45:39.603185 21724 raft_consensus.cc:374] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:39.603255 21724 raft_consensus.cc:729] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f802e0eabf9d4b82a95d24237795bd06, State: Initialized, Role: FOLLOWER
I20260924 04:45:39.603734 21724 consensus_queue.cc:260] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f802e0eabf9d4b82a95d24237795bd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260924 04:45:39.603869 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:39.603912 21724 raft_consensus.cc:388] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:39.603986 21724 raft_consensus.cc:482] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:39.604082 21724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:39.604977 21724 raft_consensus.cc:504] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f802e0eabf9d4b82a95d24237795bd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260924 04:45:39.605347 21724 leader_election.cc:302] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f802e0eabf9d4b82a95d24237795bd06; no voters:
I20260924 04:45:39.605674 21724 leader_election.cc:288] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:39.605782 21733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:39.605993 21733 raft_consensus.cc:686] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [term 1 LEADER]: Becoming Leader. State: Replica: f802e0eabf9d4b82a95d24237795bd06, State: Running, Role: LEADER
I20260924 04:45:39.606436 21733 consensus_queue.cc:237] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f802e0eabf9d4b82a95d24237795bd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260924 04:45:39.606990 21724 sys_catalog.cc:564] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:39.608278 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f802e0eabf9d4b82a95d24237795bd06. Latest consensus state: current_term: 1 leader_uuid: "f802e0eabf9d4b82a95d24237795bd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f802e0eabf9d4b82a95d24237795bd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } } }
I20260924 04:45:39.608307 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f802e0eabf9d4b82a95d24237795bd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f802e0eabf9d4b82a95d24237795bd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } } }
I20260924 04:45:39.608404 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:39.608404 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:39.608801 21744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:39.612147 21744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:39.617177 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:39.606089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:40563 (local address 127.0.0.1:36633)
0924 04:45:39.606189 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:39.606193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:39.606559 (+ 366us) server_negotiation.cc:407] Connection header received
0924 04:45:39.606700 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:39.606704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:39.606755 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:39.606841 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:39.608855 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:39.609576 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:39.611011 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:39.611411 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:39.611524 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:39.613299 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:39.613303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:39.613306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:39.613330 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:39.615088 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:39.615676 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:39.615680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:39.615682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:39.615779 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:39.616153 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:39.616156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:39.616159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:39.616384 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:39.616533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:39.616917 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 04:45:39.617026 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:45:39.618746 21744 catalog_manager.cc:1347] Generated new cluster ID: 6cf1c5d4b5a141d6a2f836fc2698ea9b
I20260924 04:45:39.618813 21744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:39.631498 21744 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:39.632546 21744 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:39.640196 21744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f802e0eabf9d4b82a95d24237795bd06: Generated new TSK 0
I20260924 04:45:39.641001 21744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:39.653036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21656
W20260924 04:45:39.658600 21641 connection.cc:535] client connection to 127.0.0.1:36633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36633): Connection reset by peer (error 104)
2026-09-24T04:45:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:39 dist-test-slave-npg2 krb5kdc[21755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:39 dist-test-slave-npg2 krb5kdc[21755](info): set up 2 sockets
Sep 24 04:45:39 dist-test-slave-npg2 krb5kdc[21755](info): commencing operation
krb5kdc: starting...
W20260924 04:45:40.699432 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:40 dist-test-slave-npg2 krb5kdc[21755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:40.826992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:40.940828 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:40.941119 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:40.941162 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:40.945324 21768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:40.945393 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:40.945413 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:40.945429 21768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:40.945446 21768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:40.950786 21768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:40.951949 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:40.953017 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:40.962241 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:40.962250 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:40.962322 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:40.962250 21768 server_base.cc:1034] running on GCE node
I20260924 04:45:40.963184 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:40.964416 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:40.965610 21768 hybrid_clock.cc:648] HybridClock initialized: now 1790225140965581 us; error 45 us; skew 500 ppm
Sep 24 04:45:40 dist-test-slave-npg2 krb5kdc[21755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:40.968461 21768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:40.969417 21768 webserver.cc:460] Webserver started at http://127.0.0.1:46113/ using document root <none> and password file <none>
I20260924 04:45:40.969990 21768 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:40.970085 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:40.970323 21768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:40.972447 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "fc205ae576f847a489d7dd1a6b3b0fc2"
format_stamp: "Formatted at 2026-09-24 04:45:40 on dist-test-slave-npg2"
server_key: "424652656b9d97df75d502306770186f"
server_key_iv: "19401d0d6d1fafc1c6d2586b6cc72499"
server_key_version: "encryptionkey@0"
I20260924 04:45:40.973034 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc205ae576f847a489d7dd1a6b3b0fc2"
format_stamp: "Formatted at 2026-09-24 04:45:40 on dist-test-slave-npg2"
server_key: "424652656b9d97df75d502306770186f"
server_key_iv: "19401d0d6d1fafc1c6d2586b6cc72499"
server_key_version: "encryptionkey@0"
I20260924 04:45:40.976825 21768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:45:40.979333 21783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:40.980619 21768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:45:40.980762 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "fc205ae576f847a489d7dd1a6b3b0fc2"
format_stamp: "Formatted at 2026-09-24 04:45:40 on dist-test-slave-npg2"
server_key: "424652656b9d97df75d502306770186f"
server_key_iv: "19401d0d6d1fafc1c6d2586b6cc72499"
server_key_version: "encryptionkey@0"
I20260924 04:45:40.980916 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:41.001081 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:41.004902 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:41.005153 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:41.013196 21768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44113
I20260924 04:45:41.013234 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44113 every 8 connection(s)
I20260924 04:45:41.014314 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:41.017462 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:41.023347 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2: Bootstrap starting.
I20260924 04:45:41.023670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21768
I20260924 04:45:41.023809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:41.024119 392 external_mini_cluster.cc:1428] Setting key 686c784f41b7bdf55fff281a4d5a3245
I20260924 04:45:41.026243 21836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:41.026995 21836 log.cc:824] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:41.028944 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2: No bootstrap required, opened a new log
I20260924 04:45:41.033376 21836 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260924 04:45:41.033624 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:41.025533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:60331 (local address 127.0.0.1:44113)
0924 04:45:41.025988 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:41.026003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:41.026038 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:45:41.026744 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:41.026765 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:41.027204 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:41.027557 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:41.028030 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:41.028996 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:41.030229 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:41.030687 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:41.030814 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:41.031840 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:41.031845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:41.031858 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:41.032247 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:41.032299 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:41.032322 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:41.032437 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:41.032694 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:45:41.032857 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":144,"threads_started":1}
I20260924 04:45:41.033646 21836 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:41.033746 21836 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc205ae576f847a489d7dd1a6b3b0fc2, State: Initialized, Role: FOLLOWER
I20260924 04:45:41.034286 21836 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260924 04:45:41.034464 21836 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:41.034528 21836 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:41.034622 21836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:41.035722 21836 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260924 04:45:41.036096 21836 leader_election.cc:302] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc205ae576f847a489d7dd1a6b3b0fc2; no voters:
I20260924 04:45:41.036544 21836 leader_election.cc:288] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:41.036788 21841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:41.037057 21841 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [term 1 LEADER]: Becoming Leader. State: Replica: fc205ae576f847a489d7dd1a6b3b0fc2, State: Running, Role: LEADER
I20260924 04:45:41.037225 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:41.037511 21841 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260924 04:45:41.037930 21836 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:41.039068 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc205ae576f847a489d7dd1a6b3b0fc2. Latest consensus state: current_term: 1 leader_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } } }
I20260924 04:45:41.039206 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:41.039479 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc205ae576f847a489d7dd1a6b3b0fc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } } }
I20260924 04:45:41.039651 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:41.039665 21854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:41.043056 21854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:41.045493 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:41.040200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:33899 (local address 127.0.0.1:44113)
0924 04:45:41.040370 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:41.040375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:41.040822 (+ 447us) server_negotiation.cc:407] Connection header received
0924 04:45:41.041036 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:41.041040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:41.041093 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:41.041197 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:41.041569 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:41.042317 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:41.043480 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:41.043821 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:41.043930 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:41.044939 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:41.044943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:41.044945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:41.044961 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:41.044991 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:41.045005 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:41.045097 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:41.045324 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 04:45:41.045363 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:45:41.049242 21854 catalog_manager.cc:1347] Generated new cluster ID: 5f14cf2d74f44006bb27ebc3da2880fb
I20260924 04:45:41.049329 21854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:41.067502 21854 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:41.068542 21854 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:41.077268 21854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc205ae576f847a489d7dd1a6b3b0fc2: Generated new TSK 0
I20260924 04:45:41.078068 21854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:41.087800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21768
W20260924 04:45:41.093483 21753 connection.cc:535] client connection to 127.0.0.1:44113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44113): Connection reset by peer (error 104)
2026-09-24T04:45:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:41 dist-test-slave-npg2 krb5kdc[21867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:41 dist-test-slave-npg2 krb5kdc[21867](info): set up 2 sockets
Sep 24 04:45:41 dist-test-slave-npg2 krb5kdc[21867](info): commencing operation
krb5kdc: starting...
W20260924 04:45:42.143437 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:42 dist-test-slave-npg2 krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:45:42.271283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:42.383282 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:42.383597 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:42.383672 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:42.387812 21880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:42.387905 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:42.387939 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:42.387979 21880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:42.388015 21880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:42.393452 21880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:42.394641 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:42.395742 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:42.404369 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:42.404394 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:42.404394 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:42.404467 21880 server_base.cc:1034] running on GCE node
I20260924 04:45:42.405169 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:42.406328 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:42.407495 21880 hybrid_clock.cc:648] HybridClock initialized: now 1790225142407478 us; error 33 us; skew 500 ppm
Sep 24 04:45:42 dist-test-slave-npg2 krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:42.410308 21880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:45:42.411293 21880 webserver.cc:460] Webserver started at http://127.0.0.1:42759/ using document root <none> and password file <none>
I20260924 04:45:42.411875 21880 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:42.411947 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:42.412178 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:42.414326 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "6cebd86bcf484e8dabd779f003eb7733"
format_stamp: "Formatted at 2026-09-24 04:45:42 on dist-test-slave-npg2"
server_key: "3201bb8cd09fd1642d73f0171018cbaf"
server_key_iv: "3e980e3267af0111be413b1616618b03"
server_key_version: "encryptionkey@0"
I20260924 04:45:42.414920 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "6cebd86bcf484e8dabd779f003eb7733"
format_stamp: "Formatted at 2026-09-24 04:45:42 on dist-test-slave-npg2"
server_key: "3201bb8cd09fd1642d73f0171018cbaf"
server_key_iv: "3e980e3267af0111be413b1616618b03"
server_key_version: "encryptionkey@0"
I20260924 04:45:42.418864 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:45:42.421507 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:42.422914 21880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:45:42.423219 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "6cebd86bcf484e8dabd779f003eb7733"
format_stamp: "Formatted at 2026-09-24 04:45:42 on dist-test-slave-npg2"
server_key: "3201bb8cd09fd1642d73f0171018cbaf"
server_key_iv: "3e980e3267af0111be413b1616618b03"
server_key_version: "encryptionkey@0"
I20260924 04:45:42.423349 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:42.428315 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:42.431421 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:42.431612 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:42.438963 21880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43539
I20260924 04:45:42.438974 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43539 every 8 connection(s)
I20260924 04:45:42.439981 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:42.442898 21948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:42.447932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21880
I20260924 04:45:42.448182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:42.448501 392 external_mini_cluster.cc:1428] Setting key 182b91a6fab5fb4e0759da3d3a32e185
I20260924 04:45:42.448836 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733: Bootstrap starting.
I20260924 04:45:42.451542 21948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:42.452416 21948 log.cc:824] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:42.454479 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:42.449999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:52039 (local address 127.0.0.1:43539)
0924 04:45:42.450475 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:42.450491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:42.450528 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:45:42.451361 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:42.451383 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:42.451746 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:42.452059 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:42.453002 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:42.453008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:42.453025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:42.453038 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:42.453108 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:42.453140 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:42.453281 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:42.453488 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:45:42.453680 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":164,"threads_started":1}
I20260924 04:45:42.454864 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733: No bootstrap required, opened a new log
I20260924 04:45:42.457924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:42.459807 21948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cebd86bcf484e8dabd779f003eb7733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260924 04:45:42.460152 21948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:42.460316 21948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cebd86bcf484e8dabd779f003eb7733, State: Initialized, Role: FOLLOWER
I20260924 04:45:42.460922 21948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cebd86bcf484e8dabd779f003eb7733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260924 04:45:42.461107 21948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:42.461175 21948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:42.461294 21948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:42.462036 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:42.460222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:35589 (local address 127.0.0.1:43539)
0924 04:45:42.460346 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:42.460350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:42.460465 (+ 115us) server_negotiation.cc:407] Connection header received
0924 04:45:42.460603 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:42.460608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:42.460651 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:42.460744 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:42.461627 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:42.461631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:42.461633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:42.461646 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:42.461678 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:42.461695 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:42.461793 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:42.461864 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:45:42.461906 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:42.462972 21948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cebd86bcf484e8dabd779f003eb7733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260924 04:45:42.463441 21948 leader_election.cc:302] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cebd86bcf484e8dabd779f003eb7733; no voters:
I20260924 04:45:42.463814 21948 leader_election.cc:288] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:42.463967 21958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:42.464272 21958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [term 1 LEADER]: Becoming Leader. State: Replica: 6cebd86bcf484e8dabd779f003eb7733, State: Running, Role: LEADER
I20260924 04:45:42.464691 21958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cebd86bcf484e8dabd779f003eb7733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260924 04:45:42.465451 21948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:42.466575 21960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cebd86bcf484e8dabd779f003eb7733. Latest consensus state: current_term: 1 leader_uuid: "6cebd86bcf484e8dabd779f003eb7733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cebd86bcf484e8dabd779f003eb7733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } } }
I20260924 04:45:42.466706 21960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:42.466854 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cebd86bcf484e8dabd779f003eb7733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cebd86bcf484e8dabd779f003eb7733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } } }
I20260924 04:45:42.467051 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733 [sys.catalog]: This master's current role is: LEADER
W20260924 04:45:42.471028 21974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:42.471146 21974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:42.471232 21967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:42.472155 21967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:42.478554 21967 catalog_manager.cc:1347] Generated new cluster ID: 19714755bce24877957bec594b83eeab
I20260924 04:45:42.478634 21967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:42.489475 21967 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:42.489899 21967 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:42.495782 21967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cebd86bcf484e8dabd779f003eb7733: Generated new TSK 0
I20260924 04:45:42.496592 21967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:42.503402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21880
2026-09-24T04:45:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:42 dist-test-slave-npg2 krb5kdc[21979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:42 dist-test-slave-npg2 krb5kdc[21979](info): set up 2 sockets
Sep 24 04:45:42 dist-test-slave-npg2 krb5kdc[21979](info): commencing operation
krb5kdc: starting...
W20260924 04:45:43.567572 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:43 dist-test-slave-npg2 krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:43.699082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:43.824168 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:43.824522 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:43.824610 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:43.829246 21992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:43.829334 21992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:43.829356 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:43.829375 21992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:43.829392 21992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:43.835114 21992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:43.836303 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:43.837364 21992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:43.846382 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:43.846341 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:43.846341 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:43.846673 21992 server_base.cc:1034] running on GCE node
I20260924 04:45:43.847378 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:43.848613 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:43.849844 21992 hybrid_clock.cc:648] HybridClock initialized: now 1790225143849830 us; error 30 us; skew 500 ppm
Sep 24 04:45:43 dist-test-slave-npg2 krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:43.852999 21992 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:43.854051 21992 webserver.cc:460] Webserver started at http://127.0.98.62:46481/ using document root <none> and password file <none>
I20260924 04:45:43.854705 21992 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:43.854797 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:43.855041 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:43.857313 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "19a41f4865da4b779652375385dafba9"
format_stamp: "Formatted at 2026-09-24 04:45:43 on dist-test-slave-npg2"
server_key: "76fc418c69fb35db7e3d125a08da83ef"
server_key_iv: "5aec07cd536134b1bdafb7eebc4c8aad"
server_key_version: "encryptionkey@0"
I20260924 04:45:43.857947 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "19a41f4865da4b779652375385dafba9"
format_stamp: "Formatted at 2026-09-24 04:45:43 on dist-test-slave-npg2"
server_key: "76fc418c69fb35db7e3d125a08da83ef"
server_key_iv: "5aec07cd536134b1bdafb7eebc4c8aad"
server_key_version: "encryptionkey@0"
I20260924 04:45:43.862002 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:43.864737 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:43.866074 21992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:45:43.866227 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "19a41f4865da4b779652375385dafba9"
format_stamp: "Formatted at 2026-09-24 04:45:43 on dist-test-slave-npg2"
server_key: "76fc418c69fb35db7e3d125a08da83ef"
server_key_iv: "5aec07cd536134b1bdafb7eebc4c8aad"
server_key_version: "encryptionkey@0"
I20260924 04:45:43.866355 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:43.881292 21992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:43.884970 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:43.885198 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:43.892807 21992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40971
I20260924 04:45:43.892886 22059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40971 every 8 connection(s)
I20260924 04:45:43.893901 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:43.896129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 21992
I20260924 04:45:43.896240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:43.896545 392 external_mini_cluster.cc:1428] Setting key 5cd66ba643d11ff15417387022f0a9c5
I20260924 04:45:43.897096 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:43.903661 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9: Bootstrap starting.
Sep 24 04:45:43 dist-test-slave-npg2 krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:43.906006 22060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:43.906821 22060 log.cc:824] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:43.908946 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9: No bootstrap required, opened a new log
I20260924 04:45:43.911713 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:43.898149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49751 (local address 127.0.98.62:40971)
0924 04:45:43.898644 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:43.898667 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:43.898708 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:45:43.899526 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:43.899545 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:43.899902 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:43.900247 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:43.901088 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:43.902103 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:43.902968 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:43.903529 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:43.903707 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:43.906316 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:43.906321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:43.906336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:43.906373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:43.908503 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:43.909080 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:43.909091 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:43.909096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:43.909239 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:43.909605 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:43.909610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:43.909616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:43.910033 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:43.910255 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:43.910585 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 04:45:43.910860 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":155,"threads_started":1}
I20260924 04:45:43.912933 22060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a41f4865da4b779652375385dafba9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:45:43.913210 22060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:43.913259 22060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a41f4865da4b779652375385dafba9, State: Initialized, Role: FOLLOWER
I20260924 04:45:43.913753 22060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a41f4865da4b779652375385dafba9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:45:43.913910 22060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:43.913998 22060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:43.914135 22060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:43.915040 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:43.915446 22060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a41f4865da4b779652375385dafba9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:45:43.915798 22060 leader_election.cc:302] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a41f4865da4b779652375385dafba9; no voters:
I20260924 04:45:43.916143 22060 leader_election.cc:288] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:43.916222 22065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:43.916476 22065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [term 1 LEADER]: Becoming Leader. State: Replica: 19a41f4865da4b779652375385dafba9, State: Running, Role: LEADER
I20260924 04:45:43.916819 22065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a41f4865da4b779652375385dafba9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:45:43.917500 22060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:43.918022 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a41f4865da4b779652375385dafba9. Latest consensus state: current_term: 1 leader_uuid: "19a41f4865da4b779652375385dafba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a41f4865da4b779652375385dafba9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } } }
I20260924 04:45:43.918174 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:43.919293 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a41f4865da4b779652375385dafba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a41f4865da4b779652375385dafba9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } } }
I20260924 04:45:43.919390 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:43.919451 22074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:43.922719 22074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:43.926750 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:43.917233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60067 (local address 127.0.98.62:40971)
0924 04:45:43.917363 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:43.917367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:43.917641 (+ 274us) server_negotiation.cc:407] Connection header received
0924 04:45:43.917837 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:43.917840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:43.917883 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:43.917945 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:43.918976 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:43.919768 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:43.920810 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:43.921147 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:43.921259 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:43.922938 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:43.922942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:43.922945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:43.922970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:43.924682 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:43.925387 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:43.925389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:43.925391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:43.925470 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:43.925853 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:43.925856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:43.925857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:43.926037 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:43.926158 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:43.926460 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 04:45:43.926543 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260924 04:45:43.928804 22074 catalog_manager.cc:1347] Generated new cluster ID: f805486bbbfa4a9ca0e325ed7cbdb6f1
I20260924 04:45:43.928879 22074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:43.939762 22074 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:43.940941 22074 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:43.946719 22074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a41f4865da4b779652375385dafba9: Generated new TSK 0
I20260924 04:45:43.947510 22074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:43.954701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 21992
W20260924 04:45:43.959638 21977 connection.cc:535] client connection to 127.0.98.62:40971 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40971): Connection reset by peer (error 104)
2026-09-24T04:45:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:44 dist-test-slave-npg2 krb5kdc[22091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:44 dist-test-slave-npg2 krb5kdc[22091](info): set up 2 sockets
Sep 24 04:45:44 dist-test-slave-npg2 krb5kdc[22091](info): commencing operation
krb5kdc: starting...
W20260924 04:45:45.007658 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:45 dist-test-slave-npg2 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:45.134621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44425
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:45.248535 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:45.248905 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:45.248997 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:45.253191 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:45.253314 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:45.253356 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:45.253394 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:45.253427 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:45.259035 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:45.260279 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:45.261394 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:45.271112 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:45.271112 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:45.271101 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:45.271440 22104 server_base.cc:1034] running on GCE node
I20260924 04:45:45.272159 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:45.273677 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:45.274919 22104 hybrid_clock.cc:648] HybridClock initialized: now 1790225145274898 us; error 35 us; skew 500 ppm
Sep 24 04:45:45 dist-test-slave-npg2 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:45.284480 22104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:45.285519 22104 webserver.cc:460] Webserver started at http://127.0.98.62:41143/ using document root <none> and password file <none>
I20260924 04:45:45.286161 22104 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:45.286268 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:45.286514 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:45.288719 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "5195dfd942144252a926f690195e042f"
format_stamp: "Formatted at 2026-09-24 04:45:45 on dist-test-slave-npg2"
server_key: "11d99d625ffc0830b486e863be8a42ce"
server_key_iv: "23fcc490f8ced72a40fa66a560810398"
server_key_version: "encryptionkey@0"
I20260924 04:45:45.289318 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "5195dfd942144252a926f690195e042f"
format_stamp: "Formatted at 2026-09-24 04:45:45 on dist-test-slave-npg2"
server_key: "11d99d625ffc0830b486e863be8a42ce"
server_key_iv: "23fcc490f8ced72a40fa66a560810398"
server_key_version: "encryptionkey@0"
I20260924 04:45:45.293159 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:45.295830 22119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:45.297047 22104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:45.297191 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "5195dfd942144252a926f690195e042f"
format_stamp: "Formatted at 2026-09-24 04:45:45 on dist-test-slave-npg2"
server_key: "11d99d625ffc0830b486e863be8a42ce"
server_key_iv: "23fcc490f8ced72a40fa66a560810398"
server_key_version: "encryptionkey@0"
I20260924 04:45:45.297327 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:45.333680 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:45.337222 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:45.337464 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:45.345227 22104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44425
I20260924 04:45:45.345255 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44425 every 8 connection(s)
I20260924 04:45:45.346280 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:45.349311 22172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:45.352303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22104
I20260924 04:45:45.352463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:45.352756 392 external_mini_cluster.cc:1428] Setting key 3bf3b74875d6221a9eacc24994a068e4
I20260924 04:45:45.355577 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f: Bootstrap starting.
I20260924 04:45:45.358302 22172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:45.359278 22172 log.cc:824] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:45.361764 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f: No bootstrap required, opened a new log
I20260924 04:45:45.362679 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:45.354163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58333 (local address 127.0.98.62:44425)
0924 04:45:45.354624 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:45.354640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:45.354675 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:45:45.355368 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:45.355388 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:45.355811 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:45.356117 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:45.356446 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:45.357448 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:45.358813 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:45.359454 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:45.359582 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:45.360655 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:45.360661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:45.360675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:45.361174 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:45.361229 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:45.361255 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:45.361377 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:45.361623 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:45:45.361910 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":157,"threads_started":1}
I20260924 04:45:45.365828 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:45.365747 22172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5195dfd942144252a926f690195e042f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260924 04:45:45.366046 22172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:45.366101 22172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5195dfd942144252a926f690195e042f, State: Initialized, Role: FOLLOWER
I20260924 04:45:45.366598 22172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5195dfd942144252a926f690195e042f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260924 04:45:45.366776 22172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:45.366847 22172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:45.366947 22172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:45.368248 22172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5195dfd942144252a926f690195e042f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260924 04:45:45.368605 22172 leader_election.cc:302] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5195dfd942144252a926f690195e042f; no voters:
I20260924 04:45:45.368854 22172 leader_election.cc:288] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:45.368954 22182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:45.369200 22182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [term 1 LEADER]: Becoming Leader. State: Replica: 5195dfd942144252a926f690195e042f, State: Running, Role: LEADER
I20260924 04:45:45.369627 22182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5195dfd942144252a926f690195e042f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260924 04:45:45.370144 22172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:45.371335 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5195dfd942144252a926f690195e042f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5195dfd942144252a926f690195e042f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } } }
I20260924 04:45:45.371474 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:45.371891 22191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:45.371893 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5195dfd942144252a926f690195e042f. Latest consensus state: current_term: 1 leader_uuid: "5195dfd942144252a926f690195e042f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5195dfd942144252a926f690195e042f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } } }
I20260924 04:45:45.372128 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:45.375294 22191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:45.377305 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:45.368161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47985 (local address 127.0.98.62:44425)
0924 04:45:45.368307 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:45.368312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:45.372578 (+ 4266us) server_negotiation.cc:407] Connection header received
0924 04:45:45.372804 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:45.372807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:45.372868 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:45.372925 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:45.373347 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:45.374159 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:45.375364 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:45.375677 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:45.375786 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:45.376779 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:45.376782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:45.376785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:45.376801 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:45.376828 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:45.376841 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:45.376944 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:45.377125 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 04:45:45.377160 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:45:45.381862 22191 catalog_manager.cc:1347] Generated new cluster ID: 56e514a4b44d4a8ebb7fbbf1c7395a50
I20260924 04:45:45.381968 22191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:45.388724 22191 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:45.389729 22191 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:45.398622 22191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5195dfd942144252a926f690195e042f: Generated new TSK 0
I20260924 04:45:45.399467 22191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:45.404377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22104
W20260924 04:45:45.410255 22089 connection.cc:535] client connection to 127.0.98.62:44425 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44425): Connection reset by peer (error 104)
2026-09-24T04:45:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:45 dist-test-slave-npg2 krb5kdc[22203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:45 dist-test-slave-npg2 krb5kdc[22203](info): set up 2 sockets
Sep 24 04:45:45 dist-test-slave-npg2 krb5kdc[22203](info): commencing operation
krb5kdc: starting...
W20260924 04:45:46.451593 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:46 dist-test-slave-npg2 krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:46.583619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:46.699785 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:46.700109 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:46.700184 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:46.704288 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:46.704383 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:46.704421 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:46.704479 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:46.704519 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:46.710024 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:46.711324 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:46.712379 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:46.721560 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:46.721554 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:46.721896 22216 server_base.cc:1034] running on GCE node
W20260924 04:45:46.721560 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:46.722579 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:46.723861 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:46.725083 22216 hybrid_clock.cc:648] HybridClock initialized: now 1790225146725046 us; error 54 us; skew 500 ppm
Sep 24 04:45:46 dist-test-slave-npg2 krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:46.728441 22216 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:46.729533 22216 webserver.cc:460] Webserver started at http://127.0.98.62:46045/ using document root <none> and password file <none>
I20260924 04:45:46.730216 22216 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:46.730324 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:46.730590 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:46.733104 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "f2f0022d4cbf4569a2b189a927ed79f2"
format_stamp: "Formatted at 2026-09-24 04:45:46 on dist-test-slave-npg2"
server_key: "de04583c3bd9eab947f9d38d80ef81b3"
server_key_iv: "8b82bdb54136edb516dc8867b20c7ad0"
server_key_version: "encryptionkey@0"
I20260924 04:45:46.733819 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2f0022d4cbf4569a2b189a927ed79f2"
format_stamp: "Formatted at 2026-09-24 04:45:46 on dist-test-slave-npg2"
server_key: "de04583c3bd9eab947f9d38d80ef81b3"
server_key_iv: "8b82bdb54136edb516dc8867b20c7ad0"
server_key_version: "encryptionkey@0"
I20260924 04:45:46.738322 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 04:45:46.741269 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:46.742637 22216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:45:46.742821 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "f2f0022d4cbf4569a2b189a927ed79f2"
format_stamp: "Formatted at 2026-09-24 04:45:46 on dist-test-slave-npg2"
server_key: "de04583c3bd9eab947f9d38d80ef81b3"
server_key_iv: "8b82bdb54136edb516dc8867b20c7ad0"
server_key_version: "encryptionkey@0"
I20260924 04:45:46.742955 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:46.747848 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:46.751519 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:46.751732 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:46.758975 22216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41591
I20260924 04:45:46.758987 22283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41591 every 8 connection(s)
I20260924 04:45:46.760012 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:46.760597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22216
I20260924 04:45:46.760766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:46.761042 392 external_mini_cluster.cc:1428] Setting key f42e721611f3c0936dd3f9a7aac5ab99
I20260924 04:45:46.763224 22284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:46.766554 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:46.762517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48841 (local address 127.0.98.62:41591)
0924 04:45:46.762933 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:46.762956 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:46.762992 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:45:46.763745 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:46.763764 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:46.764062 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:46.764354 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:46.765269 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:46.765274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:46.765287 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:46.765297 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:46.765352 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:46.765371 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:46.765490 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:46.765603 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 04:45:46.765773 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":150,"threads_started":1}
I20260924 04:45:46.769699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:46.770287 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2: Bootstrap starting.
I20260924 04:45:46.772895 22284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:46.773630 22284 log.cc:824] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:46.773846 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:46.771930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49205 (local address 127.0.98.62:41591)
0924 04:45:46.772063 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:46.772068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:46.772235 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:45:46.772356 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:46.772359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:46.772388 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:46.772485 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:46.773331 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:46.773335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:46.773337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:46.773348 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:46.773374 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:46.773388 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:46.773487 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:46.773581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:45:46.773622 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:45:46.775954 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2: No bootstrap required, opened a new log
I20260924 04:45:46.779915 22284 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260924 04:45:46.780146 22284 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:46.780227 22284 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f0022d4cbf4569a2b189a927ed79f2, State: Initialized, Role: FOLLOWER
I20260924 04:45:46.780664 22284 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260924 04:45:46.780807 22284 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:46.780875 22284 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:46.780997 22284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:46.781975 22284 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260924 04:45:46.782315 22284 leader_election.cc:302] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f0022d4cbf4569a2b189a927ed79f2; no voters:
I20260924 04:45:46.782649 22284 leader_election.cc:288] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:46.782769 22294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:46.783025 22294 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [term 1 LEADER]: Becoming Leader. State: Replica: f2f0022d4cbf4569a2b189a927ed79f2, State: Running, Role: LEADER
I20260924 04:45:46.783463 22294 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260924 04:45:46.783898 22284 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:46.785152 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f0022d4cbf4569a2b189a927ed79f2. Latest consensus state: current_term: 1 leader_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } } }
I20260924 04:45:46.785378 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:46.785152 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0022d4cbf4569a2b189a927ed79f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } } }
I20260924 04:45:46.785503 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2 [sys.catalog]: This master's current role is: LEADER
W20260924 04:45:46.788650 22309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:46.788761 22309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:46.788836 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:46.789624 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:46.795771 22308 catalog_manager.cc:1347] Generated new cluster ID: 578db9a7ed3d4d449bc3d8d0c7c84b7c
I20260924 04:45:46.795892 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:46.808259 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:46.808753 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:46.814755 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f0022d4cbf4569a2b189a927ed79f2: Generated new TSK 0
I20260924 04:45:46.815512 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:46.821424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22216
2026-09-24T04:45:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:46 dist-test-slave-npg2 krb5kdc[22315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:46 dist-test-slave-npg2 krb5kdc[22315](info): set up 2 sockets
Sep 24 04:45:46 dist-test-slave-npg2 krb5kdc[22315](info): commencing operation
krb5kdc: starting...
W20260924 04:45:47.867475 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:47 dist-test-slave-npg2 krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:47.994652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:48.107214 22328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:48.107577 22328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:48.107661 22328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:48.111743 22328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:48.111858 22328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:48.111899 22328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:48.111961 22328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:48.111999 22328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:48.117482 22328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:48.118723 22328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:48.119846 22328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:48.128520 22336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:48.128623 22333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:48.128674 22334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:48.128813 22328 server_base.cc:1034] running on GCE node
I20260924 04:45:48.129508 22328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:48.130663 22328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:48.131829 22328 hybrid_clock.cc:648] HybridClock initialized: now 1790225148131806 us; error 32 us; skew 500 ppm
Sep 24 04:45:48 dist-test-slave-npg2 krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:48.134738 22328 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:48.135783 22328 webserver.cc:460] Webserver started at http://127.0.98.62:44009/ using document root <none> and password file <none>
I20260924 04:45:48.136373 22328 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:48.136439 22328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:48.136662 22328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:48.138845 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "bea7859efdfa40929ef6b8a00d64223a"
format_stamp: "Formatted at 2026-09-24 04:45:48 on dist-test-slave-npg2"
server_key: "12f45691a03ab14cf467b2c769a93dc3"
server_key_iv: "9e73315e687be9ce6f6fdd163dec72d9"
server_key_version: "encryptionkey@0"
I20260924 04:45:48.139487 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "bea7859efdfa40929ef6b8a00d64223a"
format_stamp: "Formatted at 2026-09-24 04:45:48 on dist-test-slave-npg2"
server_key: "12f45691a03ab14cf467b2c769a93dc3"
server_key_iv: "9e73315e687be9ce6f6fdd163dec72d9"
server_key_version: "encryptionkey@0"
I20260924 04:45:48.143543 22328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:45:48.146605 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:48.148200 22328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:48.148363 22328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "bea7859efdfa40929ef6b8a00d64223a"
format_stamp: "Formatted at 2026-09-24 04:45:48 on dist-test-slave-npg2"
server_key: "12f45691a03ab14cf467b2c769a93dc3"
server_key_iv: "9e73315e687be9ce6f6fdd163dec72d9"
server_key_version: "encryptionkey@0"
I20260924 04:45:48.148511 22328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:48.170573 22328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:48.174259 22328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:48.174484 22328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:48.181782 22328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35635
I20260924 04:45:48.181936 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35635 every 8 connection(s)
I20260924 04:45:48.182829 22328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:48.185916 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:48.191434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22328
I20260924 04:45:48.191576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:48.191900 392 external_mini_cluster.cc:1428] Setting key 38de7cbb8a109b66de4d98ed438317e9
I20260924 04:45:48.192150 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a: Bootstrap starting.
I20260924 04:45:48.194730 22396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:48.195736 22396 log.cc:824] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:48.198012 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a: No bootstrap required, opened a new log
Sep 24 04:45:48 dist-test-slave-npg2 krb5kdc[22315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:48.202525 22396 raft_consensus.cc:348] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea7859efdfa40929ef6b8a00d64223a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35635 } }
I20260924 04:45:48.202797 22396 raft_consensus.cc:374] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:48.202881 22396 raft_consensus.cc:729] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea7859efdfa40929ef6b8a00d64223a, State: Initialized, Role: FOLLOWER
I20260924 04:45:48.203430 22396 consensus_queue.cc:260] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea7859efdfa40929ef6b8a00d64223a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35635 } }
I20260924 04:45:48.203593 22396 raft_consensus.cc:388] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:48.203682 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:48.203800 22396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:48.204736 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea7859efdfa40929ef6b8a00d64223a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35635 } }
I20260924 04:45:48.205114 22396 leader_election.cc:302] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bea7859efdfa40929ef6b8a00d64223a; no voters:
I20260924 04:45:48.205447 22396 leader_election.cc:288] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:48.205555 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:48.205806 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [term 1 LEADER]: Becoming Leader. State: Replica: bea7859efdfa40929ef6b8a00d64223a, State: Running, Role: LEADER
I20260924 04:45:48.206224 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea7859efdfa40929ef6b8a00d64223a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35635 } }
I20260924 04:45:48.206964 22396 sys_catalog.cc:564] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:48.207899 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:48.193260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60627 (local address 127.0.98.62:35635)
0924 04:45:48.193671 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:48.193686 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:48.193721 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:45:48.194398 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:48.194416 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:48.194817 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:48.195228 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:48.196128 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:48.197131 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:48.198298 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:48.198791 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:48.198918 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:48.201881 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:48.201886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:48.201948 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:48.201978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:48.204169 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:48.204633 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:48.204638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:48.204645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:48.204765 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:48.205056 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:48.205061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:48.205063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:48.205618 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:48.205909 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:48.206534 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 04:45:48.206844 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260924 04:45:48.208017 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bea7859efdfa40929ef6b8a00d64223a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea7859efdfa40929ef6b8a00d64223a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35635 } } }
I20260924 04:45:48.208137 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:48.208374 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bea7859efdfa40929ef6b8a00d64223a. Latest consensus state: current_term: 1 leader_uuid: "bea7859efdfa40929ef6b8a00d64223a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea7859efdfa40929ef6b8a00d64223a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35635 } } }
I20260924 04:45:48.208508 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:48.211387 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:45:48.212608 22416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:48.212720 22416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:48.212785 22409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:48.213653 22409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:48.219877 22409 catalog_manager.cc:1347] Generated new cluster ID: d9547da0871845d6b088dc5d0c4e0588
I20260924 04:45:48.220006 22409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:48.223675 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:48.213371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51525 (local address 127.0.98.62:35635)
0924 04:45:48.213482 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:48.213487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:48.215542 (+ 2055us) server_negotiation.cc:407] Connection header received
0924 04:45:48.215706 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:48.215709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:48.215765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:48.215855 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:48.216556 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:48.217140 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:48.218145 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:48.218388 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:48.218502 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:48.220136 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:48.220140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:48.220146 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:48.220172 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:48.221883 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:48.222350 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:48.222354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:48.222356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:48.222457 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:48.222737 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:48.222740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:48.222742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:48.222961 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:48.223095 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:48.223374 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 04:45:48.223507 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:45:48.238781 22409 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:48.239953 22409 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:48.252820 22409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bea7859efdfa40929ef6b8a00d64223a: Generated new TSK 0
I20260924 04:45:48.253836 22409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:48.260911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22328
2026-09-24T04:45:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:48 dist-test-slave-npg2 krb5kdc[22427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:48 dist-test-slave-npg2 krb5kdc[22427](info): set up 2 sockets
Sep 24 04:45:48 dist-test-slave-npg2 krb5kdc[22427](info): commencing operation
krb5kdc: starting...
W20260924 04:45:49.363621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:49 dist-test-slave-npg2 krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:49.498382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:49.618781 22440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:49.619074 22440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:49.619146 22440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:49.623229 22440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:49.623301 22440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:49.623320 22440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:49.623338 22440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:49.623354 22440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:49.628945 22440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:49.630115 22440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:49.631219 22440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:45:49.640605 22440 server_base.cc:1034] running on GCE node
W20260924 04:45:49.640575 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:49.640551 22445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:49.640551 22446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:49.641645 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:49.642832 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:49.644029 22440 hybrid_clock.cc:648] HybridClock initialized: now 1790225149643994 us; error 56 us; skew 500 ppm
Sep 24 04:45:49 dist-test-slave-npg2 krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:49.647020 22440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:49.648044 22440 webserver.cc:460] Webserver started at http://127.0.98.62:45691/ using document root <none> and password file <none>
I20260924 04:45:49.648593 22440 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:49.648651 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:49.648818 22440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:49.650960 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "3f12b644aafd44939f4aa49b0398e700"
format_stamp: "Formatted at 2026-09-24 04:45:49 on dist-test-slave-npg2"
server_key: "5f6314eadc6f95959c7facd018b7999a"
server_key_iv: "e87f9bbbd34ae9422abed849905f41a8"
server_key_version: "encryptionkey@0"
I20260924 04:45:49.651575 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f12b644aafd44939f4aa49b0398e700"
format_stamp: "Formatted at 2026-09-24 04:45:49 on dist-test-slave-npg2"
server_key: "5f6314eadc6f95959c7facd018b7999a"
server_key_iv: "e87f9bbbd34ae9422abed849905f41a8"
server_key_version: "encryptionkey@0"
I20260924 04:45:49.655550 22440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:45:49.658075 22455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:49.659678 22440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:49.659847 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "3f12b644aafd44939f4aa49b0398e700"
format_stamp: "Formatted at 2026-09-24 04:45:49 on dist-test-slave-npg2"
server_key: "5f6314eadc6f95959c7facd018b7999a"
server_key_iv: "e87f9bbbd34ae9422abed849905f41a8"
server_key_version: "encryptionkey@0"
I20260924 04:45:49.659974 22440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:49.679453 22440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:49.683400 22440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:49.683641 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:49.691924 22440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41411
I20260924 04:45:49.691989 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41411 every 8 connection(s)
I20260924 04:45:49.693007 22440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:49.695827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22440
I20260924 04:45:49.695938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:49.696153 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:49.696302 392 external_mini_cluster.cc:1428] Setting key 75493ec0f645bfbfb65586fa329db3b0
I20260924 04:45:49.703950 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700: Bootstrap starting.
I20260924 04:45:49.706708 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:49.706987 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:49.697988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55323 (local address 127.0.98.62:41411)
0924 04:45:49.698436 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:49.698456 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:49.698501 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:45:49.699289 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:49.699322 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:49.699778 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:49.700176 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:49.700613 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:49.702074 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:49.703016 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:49.703722 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:49.703881 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:49.704832 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:49.704839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:49.704855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:49.705419 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:49.705492 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:49.705522 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:49.705622 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:49.705770 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:45:49.705986 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260924 04:45:49.707655 22508 log.cc:824] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:49.710083 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700: No bootstrap required, opened a new log
I20260924 04:45:49.710076 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:49.714409 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f12b644aafd44939f4aa49b0398e700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41411 } }
I20260924 04:45:49.714716 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:49.714778 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f12b644aafd44939f4aa49b0398e700, State: Initialized, Role: FOLLOWER
I20260924 04:45:49.715385 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f12b644aafd44939f4aa49b0398e700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41411 } }
I20260924 04:45:49.715657 22508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:49.715754 22508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:49.715857 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:49.716883 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:49.712247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55257 (local address 127.0.98.62:41411)
0924 04:45:49.712378 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:49.712382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:49.712597 (+ 215us) server_negotiation.cc:407] Connection header received
0924 04:45:49.712735 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:49.712737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:49.712775 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:49.712842 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:49.713203 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:49.713715 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:49.714862 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:49.715264 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:49.715337 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:49.716480 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:49.716483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:49.716485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:49.716497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:49.716522 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:49.716532 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:49.716600 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:49.716710 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:45:49.716755 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:45:49.717304 22508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f12b644aafd44939f4aa49b0398e700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41411 } }
I20260924 04:45:49.717686 22508 leader_election.cc:302] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f12b644aafd44939f4aa49b0398e700; no voters:
I20260924 04:45:49.718034 22508 leader_election.cc:288] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:49.718101 22518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:49.718317 22518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [term 1 LEADER]: Becoming Leader. State: Replica: 3f12b644aafd44939f4aa49b0398e700, State: Running, Role: LEADER
I20260924 04:45:49.718636 22518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f12b644aafd44939f4aa49b0398e700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41411 } }
I20260924 04:45:49.719424 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:49.720480 22520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f12b644aafd44939f4aa49b0398e700. Latest consensus state: current_term: 1 leader_uuid: "3f12b644aafd44939f4aa49b0398e700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f12b644aafd44939f4aa49b0398e700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41411 } } }
I20260924 04:45:49.720624 22520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:49.720892 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f12b644aafd44939f4aa49b0398e700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f12b644aafd44939f4aa49b0398e700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41411 } } }
I20260924 04:45:49.721015 22527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:49.721045 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:49.724494 22527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:49.731220 22527 catalog_manager.cc:1347] Generated new cluster ID: 79a5142f6c0d4530898a3b7cf4499532
I20260924 04:45:49.731312 22527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:49.746731 22527 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:49.747900 22527 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:49.761219 22527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f12b644aafd44939f4aa49b0398e700: Generated new TSK 0
I20260924 04:45:49.762059 22527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:49.772686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22440
2026-09-24T04:45:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:49 dist-test-slave-npg2 krb5kdc[22539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:49 dist-test-slave-npg2 krb5kdc[22539](info): set up 2 sockets
Sep 24 04:45:49 dist-test-slave-npg2 krb5kdc[22539](info): commencing operation
krb5kdc: starting...
W20260924 04:45:50.835589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:50 dist-test-slave-npg2 krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:50.967171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:51.080384 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:51.080703 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:51.080746 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:51.084972 22552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:51.085041 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:51.085062 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:51.085078 22552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:51.085094 22552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:51.090555 22552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:51.091688 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:51.092774 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:51.101876 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:51.101884 22557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:51.101941 22552 server_base.cc:1034] running on GCE node
W20260924 04:45:51.101884 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:51.102820 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:51.104136 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:51.105374 22552 hybrid_clock.cc:648] HybridClock initialized: now 1790225151105347 us; error 37 us; skew 500 ppm
Sep 24 04:45:51 dist-test-slave-npg2 krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:51.114933 22552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:51.116125 22552 webserver.cc:460] Webserver started at http://127.0.98.62:33925/ using document root <none> and password file <none>
I20260924 04:45:51.116742 22552 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:51.116837 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:51.117090 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:51.119355 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "d1c29e534d9b4949b9ec988909b2afe9"
format_stamp: "Formatted at 2026-09-24 04:45:51 on dist-test-slave-npg2"
server_key: "9cd893a8ed7423c5eb82d969efafe35c"
server_key_iv: "2247b4354f9090d656119667e255ac0b"
server_key_version: "encryptionkey@0"
I20260924 04:45:51.119952 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1c29e534d9b4949b9ec988909b2afe9"
format_stamp: "Formatted at 2026-09-24 04:45:51 on dist-test-slave-npg2"
server_key: "9cd893a8ed7423c5eb82d969efafe35c"
server_key_iv: "2247b4354f9090d656119667e255ac0b"
server_key_version: "encryptionkey@0"
I20260924 04:45:51.123970 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:45:51.126578 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:51.128021 22552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:51.128180 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "d1c29e534d9b4949b9ec988909b2afe9"
format_stamp: "Formatted at 2026-09-24 04:45:51 on dist-test-slave-npg2"
server_key: "9cd893a8ed7423c5eb82d969efafe35c"
server_key_iv: "2247b4354f9090d656119667e255ac0b"
server_key_version: "encryptionkey@0"
I20260924 04:45:51.128304 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:51.133193 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:51.136559 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:51.136765 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:51.144194 22552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33307
I20260924 04:45:51.144222 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33307 every 8 connection(s)
I20260924 04:45:51.145205 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:51.148327 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:51.154287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22552
I20260924 04:45:51.154425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:51.154675 392 external_mini_cluster.cc:1428] Setting key b6f2b982c75e09efc1a8f343c585c976
I20260924 04:45:51.155681 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9: Bootstrap starting.
I20260924 04:45:51.158536 22620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:51.159395 22620 log.cc:824] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:51.161589 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9: No bootstrap required, opened a new log
I20260924 04:45:51.161854 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:51.157637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35315 (local address 127.0.98.62:33307)
0924 04:45:51.158107 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:51.158131 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:51.158167 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:45:51.158927 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:51.158947 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:51.159330 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:51.159725 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:51.160568 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:51.160573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:51.160584 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:51.160595 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:51.160648 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:51.160667 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:51.160780 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:51.160913 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:45:51.161073 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":165,"threads_started":1}
I20260924 04:45:51.164989 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:51.165875 22620 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c29e534d9b4949b9ec988909b2afe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33307 } }
I20260924 04:45:51.166096 22620 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:45:51.166155 22620 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c29e534d9b4949b9ec988909b2afe9, State: Initialized, Role: FOLLOWER
I20260924 04:45:51.166704 22620 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c29e534d9b4949b9ec988909b2afe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33307 } }
I20260924 04:45:51.166886 22620 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:45:51.166968 22620 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:45:51.167080 22620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:51.168092 22620 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c29e534d9b4949b9ec988909b2afe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33307 } }
I20260924 04:45:51.168433 22620 leader_election.cc:302] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c29e534d9b4949b9ec988909b2afe9; no voters:
I20260924 04:45:51.168787 22620 leader_election.cc:288] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:45:51.168830 22630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:51.168989 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:51.167205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46155 (local address 127.0.98.62:33307)
0924 04:45:51.167305 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:51.167310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:51.167565 (+ 255us) server_negotiation.cc:407] Connection header received
0924 04:45:51.167714 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:51.167717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:51.167750 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:51.167848 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:51.168584 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:51.168587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:51.168590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:45:51.168601 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:51.168625 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:45:51.168637 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:51.168735 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:51.168804 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:45:51.168842 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:45:51.169100 22630 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [term 1 LEADER]: Becoming Leader. State: Replica: d1c29e534d9b4949b9ec988909b2afe9, State: Running, Role: LEADER
I20260924 04:45:51.169427 22630 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c29e534d9b4949b9ec988909b2afe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33307 } }
I20260924 04:45:51.170176 22620 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:51.170644 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c29e534d9b4949b9ec988909b2afe9. Latest consensus state: current_term: 1 leader_uuid: "d1c29e534d9b4949b9ec988909b2afe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c29e534d9b4949b9ec988909b2afe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33307 } } }
I20260924 04:45:51.170760 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:51.170961 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c29e534d9b4949b9ec988909b2afe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c29e534d9b4949b9ec988909b2afe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33307 } } }
I20260924 04:45:51.171157 22634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:51.171144 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:51.174103 22634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:51.179814 22634 catalog_manager.cc:1347] Generated new cluster ID: 460491bb8ae548af805db25fcf7b455d
I20260924 04:45:51.179890 22634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:51.187029 22634 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:51.187391 22634 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:51.193815 22634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1c29e534d9b4949b9ec988909b2afe9: Generated new TSK 0
I20260924 04:45:51.194521 22634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:45:51.205475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22552
2026-09-24T04:45:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1432 ms)
[----------] 18 tests from AuthTokenIssuingTest (26021 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:51 dist-test-slave-npg2 krb5kdc[22651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:51 dist-test-slave-npg2 krb5kdc[22651](info): set up 2 sockets
Sep 24 04:45:51 dist-test-slave-npg2 krb5kdc[22651](info): commencing operation
krb5kdc: starting...
W20260924 04:45:52.271668 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:52.402135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40065,127.0.98.61:39951,127.0.98.60:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:52.517438 22664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:45:52.517843 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:52.517925 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:52.518033 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:52.522315 22664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:52.522421 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:52.522459 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:52.522497 22664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:52.522535 22664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:52.528671 22664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40065,127.0.98.61:39951,127.0.98.60:37529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:52.529861 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:52.530936 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:52.539752 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:52.539734 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:52.539734 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:52.539783 22664 server_base.cc:1034] running on GCE node
I20260924 04:45:52.540606 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:52.541770 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:52.542974 22664 hybrid_clock.cc:648] HybridClock initialized: now 1790225152542959 us; error 37 us; skew 500 ppm
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:52.545822 22664 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:52.546716 22664 webserver.cc:460] Webserver started at http://127.0.98.62:37999/ using document root <none> and password file <none>
I20260924 04:45:52.547314 22664 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:52.547387 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:52.547560 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:52.549849 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "fce67a8f79084482b65277b9b9636bda"
format_stamp: "Formatted at 2026-09-24 04:45:52 on dist-test-slave-npg2"
server_key: "053aa240138f12fcd49addfb642c0de7"
server_key_iv: "658736b68ca9155be660609230cba314"
server_key_version: "encryptionkey@0"
I20260924 04:45:52.550421 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "fce67a8f79084482b65277b9b9636bda"
format_stamp: "Formatted at 2026-09-24 04:45:52 on dist-test-slave-npg2"
server_key: "053aa240138f12fcd49addfb642c0de7"
server_key_iv: "658736b68ca9155be660609230cba314"
server_key_version: "encryptionkey@0"
I20260924 04:45:52.554463 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:52.557497 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:52.559026 22664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:45:52.559284 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "fce67a8f79084482b65277b9b9636bda"
format_stamp: "Formatted at 2026-09-24 04:45:52 on dist-test-slave-npg2"
server_key: "053aa240138f12fcd49addfb642c0de7"
server_key_iv: "658736b68ca9155be660609230cba314"
server_key_version: "encryptionkey@0"
I20260924 04:45:52.559422 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:52.577231 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:52.580904 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:52.581113 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:52.589003 22664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40065
I20260924 04:45:52.589465 22731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40065 every 8 connection(s)
I20260924 04:45:52.590127 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:52.593362 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:52.597359 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:52.599586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22664
I20260924 04:45:52.599738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:52.600044 392 external_mini_cluster.cc:1428] Setting key 2f10886a39a538d6feb0f7d14e0627cd
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:52.619518 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.599898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.0.98.62:40065)
0924 04:45:52.600503 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:52.600521 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:52.600642 (+ 121us) server_negotiation.cc:407] Connection header received
0924 04:45:52.602194 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:52.602201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:52.602473 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:52.602695 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:52.603642 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.605145 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.606383 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.607027 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.607233 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.613548 (+ 6315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:52.613555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:52.613568 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:52.613594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:52.615768 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.616461 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.616467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.616468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.616596 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.617028 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.617031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.617032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.617527 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:52.617642 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:52.618206 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 04:45:52.618561 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":483,"spinlock_wait_cycles":52992,"thread_start_us":177,"threads_started":1}
I20260924 04:45:52.619524 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.599898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.0.1:58006)
0924 04:45:52.600459 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.600523 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:52.601826 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:52.602721 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:52.602725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:52.603200 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:52.603475 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.603489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.605319 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.605322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.606247 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.606257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.607274 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.607277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.607377 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.608078 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:52.608094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:52.613297 (+ 5203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:52.615904 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.615917 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.615933 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.616302 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.616710 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.616716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.616721 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.616904 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.617663 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:52.617674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:52.617879 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 04:45:52.618099 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 04:45:52.618672 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":117,"thread_start_us":143,"threads_started":1}
I20260924 04:45:52.622287 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:52.623265 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.622912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39038)
0924 04:45:52.623066 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.623198 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 04:45:52.623471 22682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
I20260924 04:45:52.624405 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.624197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39054)
0924 04:45:52.624301 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.624347 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:45:52.624585 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39951: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:45:52.631106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39951
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40065,127.0.98.61:39951,127.0.98.60:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:45:52.673911 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } attempt: 1
I20260924 04:45:52.674934 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.674548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39056)
0924 04:45:52.674726 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.674795 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 04:45:52.675949 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.675587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39068)
0924 04:45:52.675735 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.675877 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 04:45:52.676122 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39951: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
I20260924 04:45:52.744357 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } attempt: 2
I20260924 04:45:52.745277 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.744952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39070)
0924 04:45:52.745115 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.745191 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 04:45:52.746098 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.745860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39072)
0924 04:45:52.745981 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.746038 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:45:52.746232 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39951: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39951: connect: Connection refused (error 111)
W20260924 04:45:52.748955 22739 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:45:52.749301 22739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:52.749366 22739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:52.749444 22739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:52.753721 22739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:52.753837 22739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:52.753888 22739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:52.753937 22739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:52.753978 22739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:52.759544 22739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40065,127.0.98.61:39951,127.0.98.60:37529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:52.760727 22739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:52.761826 22739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:52.771278 22744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:52.771289 22745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:52.771477 22739 server_base.cc:1034] running on GCE node
W20260924 04:45:52.771297 22747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:52.772238 22739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:52.773475 22739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:52.774674 22739 hybrid_clock.cc:648] HybridClock initialized: now 1790225152774601 us; error 91 us; skew 500 ppm
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:52.777642 22739 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:45:52.778715 22739 webserver.cc:460] Webserver started at http://127.0.98.61:34939/ using document root <none> and password file <none>
I20260924 04:45:52.779364 22739 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:52.779467 22739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:52.779707 22739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:52.781823 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data/instance:
uuid: "6397b6145b214b9e8d4a2271bc5a91a2"
format_stamp: "Formatted at 2026-09-24 04:45:52 on dist-test-slave-npg2"
server_key: "81bf094bb70df410ddd10b6f575e75f5"
server_key_iv: "82e0b131e05082a168567526ab4b46ca"
server_key_version: "encryptionkey@0"
I20260924 04:45:52.782415 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal/instance:
uuid: "6397b6145b214b9e8d4a2271bc5a91a2"
format_stamp: "Formatted at 2026-09-24 04:45:52 on dist-test-slave-npg2"
server_key: "81bf094bb70df410ddd10b6f575e75f5"
server_key_iv: "82e0b131e05082a168567526ab4b46ca"
server_key_version: "encryptionkey@0"
I20260924 04:45:52.786612 22739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:52.789695 22754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:52.791591 22739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:45:52.791766 22739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal
uuid: "6397b6145b214b9e8d4a2271bc5a91a2"
format_stamp: "Formatted at 2026-09-24 04:45:52 on dist-test-slave-npg2"
server_key: "81bf094bb70df410ddd10b6f575e75f5"
server_key_iv: "82e0b131e05082a168567526ab4b46ca"
server_key_version: "encryptionkey@0"
I20260924 04:45:52.791922 22739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:52.809828 22739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:52.813275 22739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:52.813544 22739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:52.821863 22739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39951
I20260924 04:45:52.822197 22806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39951 every 8 connection(s)
I20260924 04:45:52.822968 22739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:45:52.825999 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:52.829596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22739
I20260924 04:45:52.829698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-1/wal/instance
I20260924 04:45:52.829764 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:52.830011 392 external_mini_cluster.cc:1428] Setting key ab9523619d27de3af7fb21457d745fdf
I20260924 04:45:52.837417 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:45:52.848353 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.832041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.0.98.62:40065)
0924 04:45:52.832237 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:52.832245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:52.833103 (+ 858us) server_negotiation.cc:407] Connection header received
0924 04:45:52.834131 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:52.834134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:52.834198 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:52.834314 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:52.835648 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.836230 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.837460 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.837878 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.837985 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.843097 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:52.843106 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:52.843108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:52.843176 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:52.844823 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.845648 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.845652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.845654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.845739 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.846102 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.846107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.846110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.846357 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:52.846465 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:52.847942 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0924 04:45:52.848102 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:45:52.848703 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.832290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.0.1:58018)
0924 04:45:52.832899 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.832966 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:52.833905 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:52.834501 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:52.834508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:52.834912 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:52.835494 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.835508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.836374 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.836378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.837320 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.837330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.838104 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.838111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.838365 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.839121 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:52.839168 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:52.842933 (+ 3765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:52.845030 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.845037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.845055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.845516 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.845842 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.845846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.845848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.846002 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.846518 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:52.846532 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:52.846899 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 04:45:52.847152 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 04:45:52.847548 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":162,"threads_started":1}
I20260924 04:45:52.849560 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.837919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39086)
0924 04:45:52.838163 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.838179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:52.838274 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:52.839097 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:52.839100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:52.839335 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:52.839482 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.839490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.840854 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.840860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.841505 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.841518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.842644 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.842651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.842809 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.843373 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:52.843392 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:52.845103 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:52.847738 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.847741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.847743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.848000 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.848318 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.848320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.848322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.848416 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.849078 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:52.849081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:52.849175 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:45:52.849281 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:45:52.849419 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260924 04:45:52.850091 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.838127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.0.98.61:39951)
0924 04:45:52.838538 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:52.838548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:52.838596 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:45:52.838643 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:52.838661 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:52.838896 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:52.839090 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:52.839620 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.840661 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.841653 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.842483 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.842601 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.845247 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:52.845255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:52.845270 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:52.845300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:52.847619 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.848132 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.848138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.848140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.848219 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.848663 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.848666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.848668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.848946 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:52.849107 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:52.849792 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 04:45:52.849943 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":183,"threads_started":1}
I20260924 04:45:52.850443 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:52.852005 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:52.852712 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.852425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34820)
0924 04:45:52.852566 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.852618 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:45:52.853366 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.853113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34830)
0924 04:45:52.853250 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.853297 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 04:45:52.853489 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Sep 24 04:45:52 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:45:52.860651 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.850900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39092)
0924 04:45:52.851011 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.851026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:52.851101 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:52.851415 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:52.851418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:52.851582 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:52.851661 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.851665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.852591 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.852594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.853390 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:52.853404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.854057 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.854064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:52.854171 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.854696 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:52.854716 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:52.856859 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:52.858989 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.858992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.858993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.859256 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.859621 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:52.859623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:52.859625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:52.859704 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:52.860221 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:52.860223 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:52.860304 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 04:45:52.860383 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:45:52.860489 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:45:52.860934 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.851123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.98.61:39951)
0924 04:45:52.851236 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:52.851239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:52.851250 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:45:52.851282 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:52.851284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:52.851315 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:52.851408 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:52.851766 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.852445 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.853528 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:52.853802 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:52.853887 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:52.857048 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:52.857051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:52.857053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:52.857076 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:52.858825 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.859389 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.859414 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.859419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.859523 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:52.859829 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:52.859833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:52.859835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:52.860127 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:52.860209 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:52.860594 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 04:45:52.860737 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:45:52.861814 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:45:52.862740 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.862366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34844)
0924 04:45:52.862561 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.862670 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 04:45:52.863174 22755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:52.863620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37529
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40065,127.0.98.61:39951,127.0.98.60:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:45:52.864230 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.863962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34850)
0924 04:45:52.864074 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.864162 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:45:52.864411 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:52.884673 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 1
I20260924 04:45:52.885663 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.885275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34860)
0924 04:45:52.885487 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.885564 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 04:45:52.886569 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.886327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34872)
0924 04:45:52.886446 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.886500 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:45:52.886723 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:52.913710 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 1
I20260924 04:45:52.914790 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.914454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34886)
0924 04:45:52.914634 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.914706 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 04:45:52.915652 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.915428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34894)
0924 04:45:52.915542 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.915596 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:45:52.915781 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:52.961988 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 2
I20260924 04:45:52.962944 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.962601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34908)
0924 04:45:52.962767 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.962853 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 04:45:52.963920 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.963581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34916)
0924 04:45:52.963768 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.963836 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 04:45:52.964098 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:52.983981 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 2
I20260924 04:45:52.985010 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.984661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34932)
0924 04:45:52.984841 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.984913 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 04:45:52.985966 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:52.985694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34936)
0924 04:45:52.985834 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:52.985894 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 04:45:52.986145 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
W20260924 04:45:52.991505 22814 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:45:52.991838 22814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:52.991866 22814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:52.991911 22814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:52.996781 22814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:52.996867 22814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:52.996891 22814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:52.996910 22814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:52.996928 22814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:53.002913 22814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40065,127.0.98.61:39951,127.0.98.60:37529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:53.004144 22814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:53.005175 22814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:53.014465 22822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:53.014616 22819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:53.014469 22820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:53.014770 22814 server_base.cc:1034] running on GCE node
I20260924 04:45:53.015681 22814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:53.016983 22814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:53.018165 22814 hybrid_clock.cc:648] HybridClock initialized: now 1790225153018124 us; error 55 us; skew 500 ppm
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:53.021698 22814 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:45:53.022858 22814 webserver.cc:460] Webserver started at http://127.0.98.60:42747/ using document root <none> and password file <none>
I20260924 04:45:53.023607 22814 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:53.023713 22814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:53.023988 22814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:53.026454 22814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data/instance:
uuid: "a33da7ed4de6423e8b16576fa483bfad"
format_stamp: "Formatted at 2026-09-24 04:45:53 on dist-test-slave-npg2"
server_key: "e026502362201a07f7742b85146e049e"
server_key_iv: "ac65adf6411eff1d472568face419746"
server_key_version: "encryptionkey@0"
I20260924 04:45:53.027218 22814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal/instance:
uuid: "a33da7ed4de6423e8b16576fa483bfad"
format_stamp: "Formatted at 2026-09-24 04:45:53 on dist-test-slave-npg2"
server_key: "e026502362201a07f7742b85146e049e"
server_key_iv: "ac65adf6411eff1d472568face419746"
server_key_version: "encryptionkey@0"
I20260924 04:45:53.032642 22814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 04:45:53.035590 22829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:53.037228 22814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:45:53.037413 22814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal
uuid: "a33da7ed4de6423e8b16576fa483bfad"
format_stamp: "Formatted at 2026-09-24 04:45:53 on dist-test-slave-npg2"
server_key: "e026502362201a07f7742b85146e049e"
server_key_iv: "ac65adf6411eff1d472568face419746"
server_key_version: "encryptionkey@0"
I20260924 04:45:53.037550 22814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:53.063269 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 3
I20260924 04:45:53.064201 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.063782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34942)
0924 04:45:53.064001 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.064106 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 04:45:53.065157 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.064796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34948)
0924 04:45:53.065026 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.065085 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260924 04:45:53.065303 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:53.073275 22814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:53.077318 22814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:53.077338 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 3
I20260924 04:45:53.077564 22814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:53.078269 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.077905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34954)
0924 04:45:53.078096 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.078169 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 04:45:53.079358 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.078927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34958)
0924 04:45:53.079176 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.079235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260924 04:45:53.079571 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37529: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37529: connect: Connection refused (error 111)
I20260924 04:45:53.086205 22814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37529
I20260924 04:45:53.086532 22881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37529 every 8 connection(s)
I20260924 04:45:53.087568 22814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:45:53.090660 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:53.091939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 22814
I20260924 04:45:53.092069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224865297037-392-0/minicluster-data/master-2/wal/instance
I20260924 04:45:53.092388 392 external_mini_cluster.cc:1428] Setting key ca0c7a09480a302ddd5e01af3e442eb4
I20260924 04:45:53.095597 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:53.108448 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.097844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.0.98.62:40065)
0924 04:45:53.098049 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.098054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.098353 (+ 299us) server_negotiation.cc:407] Connection header received
0924 04:45:53.098528 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.098531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.098580 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.098671 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.099391 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.099895 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.100848 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.101209 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.101327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.104611 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.104615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.104617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.104646 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.106459 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.106918 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.106922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.106925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.107014 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.107379 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.107383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.107385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.107627 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.107730 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.108154 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.108279 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 04:45:53.108886 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.093959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.0.98.60:37529)
0924 04:45:53.094462 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.094478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.094518 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:45:53.095392 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.095414 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.095817 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.096249 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.097165 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.098666 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.099644 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.100136 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.100317 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.103092 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.103098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.103131 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.103193 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.105155 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.105787 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.105795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.105906 (+ 111us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.106062 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.106371 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.106378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.106382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.106898 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.107104 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.107521 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.107923 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260924 04:45:53.108891 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.097758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.0.1:58024)
0924 04:45:53.098225 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.098264 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:53.098408 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:53.098720 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:53.098726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:53.099081 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:53.099264 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.099270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.100016 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.100020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.100698 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.100709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.101372 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.101375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.101465 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.102591 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:53.102607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:53.104453 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:53.106579 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.106588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.106592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.106814 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.107151 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.107155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.107157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.107255 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.107749 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:53.107754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:53.107867 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:45:53.108045 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:45:53.108185 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":133,"threads_started":1}
I20260924 04:45:53.110338 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:53.120096 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.110866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39951 (local address 127.0.0.1:39104)
0924 04:45:53.110994 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.111010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:53.111086 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:53.111625 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:53.111628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:53.111823 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:53.111934 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.111939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.112942 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.112945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.113531 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.113539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.114120 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.114123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.114235 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.114714 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:53.114727 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:53.116316 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:53.118325 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.118330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.118335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.118643 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.118950 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.118953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.118955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.119048 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.119616 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:53.119621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:53.119727 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:45:53.119838 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:45:53.119971 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:45:53.120378 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.111229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.0.98.61:39951)
0924 04:45:53.111361 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.111366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.111388 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:45:53.111431 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.111434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.111489 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.111572 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.112061 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.112739 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.113652 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.114005 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.114092 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.116485 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.116488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.116490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.116517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.118192 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.118761 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.118764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.118765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.118845 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.119193 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.119197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.119199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.119509 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.119612 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.119911 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.120029 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:45:53.121129 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:53.121233 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.111353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.0.98.62:40065)
0924 04:45:53.111458 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.111462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.111501 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:45:53.111630 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.111632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.111680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.111764 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.112462 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.113043 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.113882 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.114125 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.114224 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.117210 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.117215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.117218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.117251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.119042 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.119548 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.119551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.119552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.119660 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.120178 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.120182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.120183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.120414 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.120668 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.120965 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.121085 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:45:53.129945 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.121690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34972)
0924 04:45:53.121799 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.121815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:53.121885 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:53.122122 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:53.122125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:53.122294 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:53.122357 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.122361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.123094 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.123097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.123642 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.123650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.124118 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.124121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.124216 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.124706 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:53.124721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:53.126053 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:53.128269 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.128275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.128277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.128495 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.128831 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.128835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.128838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.128941 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.129509 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:53.129514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:53.129620 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:45:53.129706 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:45:53.129812 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260924 04:45:53.130131 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.121730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.0.98.60:37529)
0924 04:45:53.121948 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.121951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.121962 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:45:53.121989 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.121991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.122024 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.122114 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.122466 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.122981 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.123760 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.124011 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.124121 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.126223 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.126226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.126228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.126250 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.128150 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.128602 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.128609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.128615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.128710 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.129057 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.129061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.129063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.129347 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.129525 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.129823 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.129947 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":48}
I20260924 04:45:53.132583 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.122428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.0.98.61:39951)
0924 04:45:53.122532 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.122536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.122602 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:45:53.122744 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.122746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.122785 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.122873 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.123548 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.124029 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.124881 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.125151 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.125263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.128357 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.128360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.128362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.128395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.130741 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.131267 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.131271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.131275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.131358 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.131675 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.131678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.131680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.131897 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.132091 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.132274 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.132376 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:45:53.134138 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: Bootstrap starting.
I20260924 04:45:53.136373 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:53.137156 22882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:53.138039 22882 log.cc:824] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:53.137753 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fce67a8f79084482b65277b9b9636bda"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58026
I20260924 04:45:53.140149 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: No bootstrap required, opened a new log
I20260924 04:45:53.143298 22882 raft_consensus.cc:348] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.143505 22882 raft_consensus.cc:729] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33da7ed4de6423e8b16576fa483bfad, State: Initialized, Role: FOLLOWER
I20260924 04:45:53.143831 22882 consensus_queue.cc:260] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.144378 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.144521 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:53.145045 22882 sys_catalog.cc:564] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:45:53.150434 22899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:53.150520 22899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:53.148928 22814 master_runner.cc:186] Error getting master registration for 127.0.98.62:40065: OK, instance_id { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" instance_seqno: 1790225152586755 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:45:53.151588 22814 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:45:53.222746 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 4
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:45:53.229460 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } attempt: 4
I20260924 04:45:53.233314 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.223600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.60:37529)
0924 04:45:53.223735 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.223740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.223759 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:45:53.223809 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.223813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.223867 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.223955 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.224381 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.225163 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.226286 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.226636 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.226749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.229345 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.229350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.229353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.229380 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.231569 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.232089 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.232092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.232094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.232175 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.232441 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.232446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.232448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.232656 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.232770 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.233048 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.233153 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:45:53.233314 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.223429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34974)
0924 04:45:53.223599 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.223616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:53.223729 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:53.223977 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:53.223980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:53.224151 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:53.224244 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.224249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.225287 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.225290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.226162 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.226172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.226766 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.226769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.226859 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.227461 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:53.227476 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:53.229158 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:53.231684 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.231688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.231690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.231978 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.232269 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.232274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.232278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.232373 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.232761 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:53.232764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:53.232870 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:45:53.232994 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:45:53.233142 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:45:53.236698 22807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: Bootstrap starting.
Sep 24 04:45:53 dist-test-slave-npg2 krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:45:53.239678 22807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:53.240594 22807 log.cc:824] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:53.241120 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.229994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37529 (local address 127.0.0.1:34982)
0924 04:45:53.230157 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:53.230176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:53.230268 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:53.231238 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:53.231242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:53.231470 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:53.231606 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.231613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.232727 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.232731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.233559 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:53.233570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.235247 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.235251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:53.235350 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.235824 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:53.235838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:53.237342 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:53.239480 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.239484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.239486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.239743 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.240076 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:53.240079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:53.240081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:53.240165 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:53.240611 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:53.240614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:53.240712 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 04:45:53.240795 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:45:53.240944 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:45:53.241804 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.230533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.0.98.60:37529)
0924 04:45:53.230909 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.230919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.230934 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:45:53.230975 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.230981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.231036 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.231200 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.231757 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.232598 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.233706 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.234083 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.234180 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.237505 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.237509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.237512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.237536 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.239260 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.239886 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.239890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.239892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.239973 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.240268 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.240270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.240272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.240500 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.241426 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.241500 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.241621 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260924 04:45:53.243619 22807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: No bootstrap required, opened a new log
I20260924 04:45:53.244938 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda: Bootstrap starting.
I20260924 04:45:53.247745 22732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:53.247745 22807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.248077 22807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6397b6145b214b9e8d4a2271bc5a91a2, State: Initialized, Role: FOLLOWER
I20260924 04:45:53.248461 22807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.248628 22732 log.cc:824] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:53.249172 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.249316 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:53.250029 22807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:53.254287 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda: No bootstrap required, opened a new log
W20260924 04:45:53.255419 22915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:53.255499 22915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:53.257506 22732 raft_consensus.cc:348] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.257777 22732 raft_consensus.cc:729] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce67a8f79084482b65277b9b9636bda, State: Initialized, Role: FOLLOWER
I20260924 04:45:53.258129 22732 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.259469 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.259737 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:53.259750 22739 master_runner.cc:186] Error getting master registration for 127.0.98.62:40065: OK, instance_id { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" instance_seqno: 1790225152586755 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:45:53.263247 22739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:45:53.263880 22732 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:53.263945 22706 raft_consensus.cc:482] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:45:53.264081 22706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:53.265412 22706 raft_consensus.cc:504] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.266419 22706 leader_election.cc:288] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [CANDIDATE]: Term 1 election: Requested vote from peers 6397b6145b214b9e8d4a2271bc5a91a2 (127.0.98.61:39951), a33da7ed4de6423e8b16576fa483bfad (127.0.98.60:37529)
I20260924 04:45:53.267436 22856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fce67a8f79084482b65277b9b9636bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a33da7ed4de6423e8b16576fa483bfad"
I20260924 04:45:53.267727 22856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:53.269152 22856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fce67a8f79084482b65277b9b9636bda in term 1.
I20260924 04:45:53.269636 22781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fce67a8f79084482b65277b9b9636bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6397b6145b214b9e8d4a2271bc5a91a2"
I20260924 04:45:53.269905 22781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:53.271238 22781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fce67a8f79084482b65277b9b9636bda in term 1.
I20260924 04:45:53.271677 22682 leader_election.cc:302] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6397b6145b214b9e8d4a2271bc5a91a2, fce67a8f79084482b65277b9b9636bda; no voters:
I20260924 04:45:53.272037 22917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:53.272274 22917 raft_consensus.cc:686] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [term 1 LEADER]: Becoming Leader. State: Replica: fce67a8f79084482b65277b9b9636bda, State: Running, Role: LEADER
I20260924 04:45:53.272639 22917 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } }
I20260924 04:45:53.275092 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce67a8f79084482b65277b9b9636bda. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.279995 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: This master's current role is: LEADER
W20260924 04:45:53.289539 22928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:53.289639 22928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:53.289698 22938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:53.290565 22938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:45:53.291657 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.275241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.0.98.61:39951)
0924 04:45:53.275364 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.275369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.279497 (+ 4128us) server_negotiation.cc:407] Connection header received
0924 04:45:53.279729 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.279732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.279780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.279833 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.282576 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.283415 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.284849 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.285162 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.285239 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.287638 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.287642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.287645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.287675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.289339 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.290142 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.290148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.290150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.290239 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.290536 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.290539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.290541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.290765 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.290868 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.291363 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.291490 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:53.294718 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.269574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.98.60:37529)
0924 04:45:53.269798 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.269803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.271494 (+ 1691us) server_negotiation.cc:407] Connection header received
0924 04:45:53.271683 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.271687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.271735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.271786 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.274375 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.275230 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.279075 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.279439 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.279543 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.283259 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.283264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.283266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.283295 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.284987 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.285876 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.285880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.285882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.285968 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.293025 (+ 7057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.293030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.293031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.293269 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.293382 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.294424 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.294561 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260924 04:45:53.295809 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:53.269472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.0.98.62:40065)
0924 04:45:53.269698 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.269703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.272386 (+ 2683us) server_negotiation.cc:407] Connection header received
0924 04:45:53.272990 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.272993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.273040 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.273088 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.276471 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.277216 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.278284 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:53.278655 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:53.278737 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:53.285522 (+ 6785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:53.285526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:53.285529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:53.285557 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:53.287294 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.292587 (+ 5293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.292591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.292593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.292690 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:53.293702 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:53.293706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:53.293708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:53.293963 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:53.294088 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:53.295521 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0924 04:45:53.295655 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260924 04:45:53.303072 22856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [term 1 FOLLOWER]: Refusing update from remote peer fce67a8f79084482b65277b9b9636bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:53.303069 22781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [term 1 FOLLOWER]: Refusing update from remote peer fce67a8f79084482b65277b9b9636bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:53.304255 22941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:53.309597 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce67a8f79084482b65277b9b9636bda. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.309763 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:53.313314 22936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260924 04:45:53.317906 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.318050 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:53.318466 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce67a8f79084482b65277b9b9636bda. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.318600 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:45:53.318779 22760 tablet.cc:2369] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:45:53.318745 22942 mvcc.cc:205] Tried to move back new op lower bound from 7332762227897331712 to 7332762227808260096. Current Snapshot: MvccSnapshot[applied={T|T < 7332762227897331712}]
I20260924 04:45:53.319525 22936 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.319666 22936 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:53.319816 22940 mvcc.cc:205] Tried to move back new op lower bound from 7332762227897331712 to 7332762227808260096. Current Snapshot: MvccSnapshot[applied={T|T < 7332762227897331712}]
I20260924 04:45:53.320851 22938 catalog_manager.cc:1347] Generated new cluster ID: bcbdc2818e634762a68a19dd96b5070b
I20260924 04:45:53.320966 22938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:53.334465 22938 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:53.335381 22936 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.335520 22936 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:53.336009 22938 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:53.337118 22945 mvcc.cc:205] Tried to move back new op lower bound from 7332762227897331712 to 7332762227808260096. Current Snapshot: MvccSnapshot[applied={T|T < 7332762227897331712}]
I20260924 04:45:53.337091 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fce67a8f79084482b65277b9b9636bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fce67a8f79084482b65277b9b9636bda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } peers { permanent_uuid: "6397b6145b214b9e8d4a2271bc5a91a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39951 } } peers { permanent_uuid: "a33da7ed4de6423e8b16576fa483bfad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37529 } } }
I20260924 04:45:53.337229 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:53.357316 22938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fce67a8f79084482b65277b9b9636bda: Generated new TSK 0
I20260924 04:45:53.358542 22938 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:45:53.390173 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.388552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.0.98.62:40065)
0924 04:45:53.388709 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.388715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.389316 (+ 601us) server_negotiation.cc:407] Connection header received
0924 04:45:53.389440 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.389443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.389495 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.389579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.390068 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58046: BlockingRecv error: recv got EOF from 127.0.0.1:58046 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 04:45:53.390163 22901 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.388427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.0.98.60:37529)
0924 04:45:53.388581 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.388586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.388929 (+ 343us) server_negotiation.cc:407] Connection header received
0924 04:45:53.389156 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.389160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.389216 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.389272 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.390010 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34998: BlockingRecv error: recv got EOF from 127.0.0.1:34998 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:45:53.391007 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.389077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.98.61:39951)
0924 04:45:53.389196 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.389202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.389620 (+ 418us) server_negotiation.cc:407] Connection header received
0924 04:45:53.389795 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.389799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.389849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.389900 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.390885 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39144: BlockingRecv error: recv got EOF from 127.0.0.1:39144 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:45:53.394173 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.393042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.98.61:39951)
0924 04:45:53.393144 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.393149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.393325 (+ 176us) server_negotiation.cc:407] Connection header received
0924 04:45:53.393474 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.393478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.393557 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.393645 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.394072 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39152: BlockingRecv error: recv got EOF from 127.0.0.1:39152 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 04:45:53.404596 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.397533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.0.98.61:39951)
0924 04:45:53.397659 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.397664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.403516 (+ 5852us) server_negotiation.cc:407] Connection header received
0924 04:45:53.403675 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.403678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.403741 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.403837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.404469 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39164: BlockingRecv error: recv got EOF from 127.0.0.1:39164 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:45:53.410131 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.408797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.98.61:39951)
0924 04:45:53.408924 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.408929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.409157 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:45:53.409309 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.409312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.409379 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.409477 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.410014 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39166: BlockingRecv error: recv got EOF from 127.0.0.1:39166 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:45:53.417832 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.416685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.0.98.61:39951)
0924 04:45:53.416813 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.416817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.417015 (+ 198us) server_negotiation.cc:407] Connection header received
0924 04:45:53.417130 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.417134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.417187 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.417283 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.417724 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39180: BlockingRecv error: recv got EOF from 127.0.0.1:39180 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:45:53.429771 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.428361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.0.98.61:39951)
0924 04:45:53.428490 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.428495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.428776 (+ 281us) server_negotiation.cc:407] Connection header received
0924 04:45:53.428921 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.428926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.428987 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.429082 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.429613 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39196: BlockingRecv error: recv got EOF from 127.0.0.1:39196 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:45:53.449699 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.448239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.98.61:39951)
0924 04:45:53.448375 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.448380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.448685 (+ 305us) server_negotiation.cc:407] Connection header received
0924 04:45:53.448870 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.448874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.448941 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.449044 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.449571 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39202: BlockingRecv error: recv got EOF from 127.0.0.1:39202 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:45:53.485299 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.484040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.0.98.61:39951)
0924 04:45:53.484227 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.484231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.484309 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:45:53.484444 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.484447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.484500 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.484581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.485140 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39212: BlockingRecv error: recv got EOF from 127.0.0.1:39212 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:45:53.553160 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.551687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.0.98.61:39951)
0924 04:45:53.551870 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.551876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.552169 (+ 293us) server_negotiation.cc:407] Connection header received
0924 04:45:53.552335 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.552338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.552392 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.552478 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.552994 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39228: BlockingRecv error: recv got EOF from 127.0.0.1:39228 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 04:45:53.684993 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.683656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.0.98.61:39951)
0924 04:45:53.683925 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.683932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.684008 (+ 76us) server_negotiation.cc:407] Connection header received
0924 04:45:53.684187 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.684190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.684249 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.684353 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.684853 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39244: BlockingRecv error: recv got EOF from 127.0.0.1:39244 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 04:45:53.944589 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:53.943394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.0.98.61:39951)
0924 04:45:53.943549 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:53.943554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:53.943590 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:45:53.943739 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:53.943741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:53.943790 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:53.943874 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:53.944407 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39260: BlockingRecv error: recv got EOF from 127.0.0.1:39260 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:45:54.152036 22899 catalog_manager.cc:1259] Loaded cluster ID: bcbdc2818e634762a68a19dd96b5070b
I20260924 04:45:54.152123 22899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: loading cluster ID for follower catalog manager: success
I20260924 04:45:54.153949 22899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: acquiring CA information for follower catalog manager: success
I20260924 04:45:54.155177 22899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a33da7ed4de6423e8b16576fa483bfad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:45:54.256886 22915 catalog_manager.cc:1259] Loaded cluster ID: bcbdc2818e634762a68a19dd96b5070b
I20260924 04:45:54.257016 22915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: loading cluster ID for follower catalog manager: success
I20260924 04:45:54.258733 22915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: acquiring CA information for follower catalog manager: success
I20260924 04:45:54.260066 22915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6397b6145b214b9e8d4a2271bc5a91a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:45:54.463517 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:54.458722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.0.98.61:39951)
0924 04:45:54.459079 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:54.459086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:54.459221 (+ 135us) server_negotiation.cc:407] Connection header received
0924 04:45:54.459440 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:54.459444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:54.459594 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:54.459730 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:54.460125 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.460755 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.461752 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.462018 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.462150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:54.462525 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:54.462999 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:54.463180 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:54.463249 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:45:54.463319 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260924 04:45:54.471218 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:54.467238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.98.60:37529)
0924 04:45:54.467505 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:54.467509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:54.467523 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:45:54.467653 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:54.467656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:54.467770 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:54.467871 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:54.468110 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.468630 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.469591 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.469819 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.469912 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:54.470198 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:54.470633 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:54.470728 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:54.470837 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:45:54.470887 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260924 04:45:54.478760 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:54.474857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.98.60:37529)
0924 04:45:54.474979 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:54.474983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:54.474996 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:45:54.475106 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:54.475110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:54.475185 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:54.475263 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:54.475535 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.476306 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.477460 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.477801 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.477904 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:54.478227 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:54.478308 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:54.478378 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:54.478544 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:45:54.478604 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:45:54.479326 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:54.475063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.0.98.61:39951)
0924 04:45:54.475239 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:54.475246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:54.475679 (+ 433us) server_negotiation.cc:407] Connection header received
0924 04:45:54.475873 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:54.475877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:54.475930 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:54.476010 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:54.476256 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.477043 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.478031 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.478368 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.478479 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:54.478701 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:54.478779 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:54.478857 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:54.479110 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:45:54.479195 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 04:45:54.479527 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:54.474609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.0.98.62:40065)
0924 04:45:54.475181 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:54.475185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:54.475199 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:45:54.475327 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:54.475330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:54.475520 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:54.475641 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:54.475919 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.476585 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.477616 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:54.477981 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:54.478118 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:54.478488 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:54.478853 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:54.479272 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:54.479308 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:45:54.479373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":70,"threads_started":1}
I20260924 04:45:54.482963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22664
I20260924 04:45:54.489660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22739
I20260924 04:45:54.496670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 22814
2026-09-24T04:45:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3293 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:45:54 dist-test-slave-npg2 krb5kdc[23032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:45:54 dist-test-slave-npg2 krb5kdc[23032](info): set up 2 sockets
Sep 24 04:45:54 dist-test-slave-npg2 krb5kdc[23032](info): commencing operation
krb5kdc: starting...
W20260924 04:45:55.555725 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/test-user.keytab.
Sep 24 04:45:55 dist-test-slave-npg2 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:45:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:45:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:45:55.688181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42459,127.0.98.61:39657,127.0.98.60:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:45:55.799875 23045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:45:55.800252 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:55.800333 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:55.800419 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:55.804543 23045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:55.804651 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:55.804688 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:55.804729 23045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:55.804764 23045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:55.810187 23045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42459,127.0.98.61:39657,127.0.98.60:38865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:55.811453 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:55.812498 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:55.821262 23051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:55.821441 23045 server_base.cc:1034] running on GCE node
W20260924 04:45:55.821457 23050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:55.821259 23053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:55.822149 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:55.823408 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:55.824623 23045 hybrid_clock.cc:648] HybridClock initialized: now 1790225155824595 us; error 40 us; skew 500 ppm
Sep 24 04:45:55 dist-test-slave-npg2 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:55.827507 23045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:45:55.828454 23045 webserver.cc:460] Webserver started at http://127.0.98.62:38541/ using document root <none> and password file <none>
I20260924 04:45:55.829046 23045 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:55.829151 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:55.829391 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:55.831529 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data/instance:
uuid: "c4e70cc6bcc146598faa05ee67fb8d08"
format_stamp: "Formatted at 2026-09-24 04:45:55 on dist-test-slave-npg2"
server_key: "35b8e7b7ade4e9dbfae5e825d8e48c37"
server_key_iv: "03c74c32fb471c986c595733999fcc9c"
server_key_version: "encryptionkey@0"
I20260924 04:45:55.832109 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4e70cc6bcc146598faa05ee67fb8d08"
format_stamp: "Formatted at 2026-09-24 04:45:55 on dist-test-slave-npg2"
server_key: "35b8e7b7ade4e9dbfae5e825d8e48c37"
server_key_iv: "03c74c32fb471c986c595733999fcc9c"
server_key_version: "encryptionkey@0"
I20260924 04:45:55.836040 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:45:55.838486 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:55.839807 23045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:45:55.839947 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal
uuid: "c4e70cc6bcc146598faa05ee67fb8d08"
format_stamp: "Formatted at 2026-09-24 04:45:55 on dist-test-slave-npg2"
server_key: "35b8e7b7ade4e9dbfae5e825d8e48c37"
server_key_iv: "03c74c32fb471c986c595733999fcc9c"
server_key_version: "encryptionkey@0"
I20260924 04:45:55.840061 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:55.855573 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:55.859491 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:55.859714 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:55.867363 23045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42459
I20260924 04:45:55.867694 23112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42459 every 8 connection(s)
I20260924 04:45:55.868466 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:45:55.871482 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:55.875231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 23045
I20260924 04:45:55.875355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-0/wal/instance
I20260924 04:45:55.875694 392 external_mini_cluster.cc:1428] Setting key 1f92cd9d87cec3f1d0cfc20ff2cea61d
I20260924 04:45:55.875494 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:45:55 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:55.894196 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:55.877791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.0.98.62:42459)
0924 04:45:55.878361 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:55.878376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:55.878469 (+ 93us) server_negotiation.cc:407] Connection header received
0924 04:45:55.880034 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:55.880039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:55.880264 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:55.880464 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:55.881667 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:55.882534 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:55.884149 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:55.884564 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:55.884730 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:55.888738 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:55.888744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:55.888756 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:55.888778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:55.890543 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:55.891169 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:55.891179 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:55.891181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:55.891285 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:55.891711 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:55.891714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:55.891715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:55.892198 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:55.892314 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:55.892989 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 04:45:55.893387 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":203,"server-negotiator.queue_time_us":448,"spinlock_wait_cycles":34560,"thread_start_us":176,"threads_started":1}
I20260924 04:45:55.894199 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:55.877790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42459 (local address 127.0.0.1:37946)
0924 04:45:55.878283 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:55.878338 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:55.879699 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:55.880530 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:55.880538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:55.881055 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:55.881471 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:55.881495 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:55.882672 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:55.882676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:55.884023 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:55.884035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:55.884752 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:55.884759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:55.884902 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:55.885552 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:55.885567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:55.888566 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:55.890652 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:55.890664 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:55.890678 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:55.891008 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:55.891396 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:55.891402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:55.891408 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:55.891570 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:55.892336 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:55.892345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:55.892556 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 04:45:55.892777 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 04:45:55.893208 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":147,"threads_started":1}
I20260924 04:45:55.897126 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:55.897953 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:55.897644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41152)
0924 04:45:55.897787 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:55.897889 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 04:45:55.898188 23064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
I20260924 04:45:55.899204 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:55.898919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41160)
0924 04:45:55.899046 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:55.899091 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 04:45:55.899348 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:45:55.906690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39657
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42459,127.0.98.61:39657,127.0.98.60:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:45:55.948654 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } attempt: 1
I20260924 04:45:55.949640 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:55.949320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41164)
0924 04:45:55.949492 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:55.949561 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 04:45:55.950457 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:55.950235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41180)
0924 04:45:55.950345 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:55.950398 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:45:55.950604 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
I20260924 04:45:56.018816 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } attempt: 2
I20260924 04:45:56.019862 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.019530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41194)
0924 04:45:56.019696 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.019771 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 04:45:56.020644 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.020409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41208)
0924 04:45:56.020536 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.020590 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:45:56.020812 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39657: connect: Connection refused (error 111)
W20260924 04:45:56.023602 23120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:45:56.023963 23120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:56.024029 23120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:56.024127 23120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:56.028476 23120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:56.028569 23120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:56.028606 23120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:56.028671 23120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:56.028707 23120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:56.034499 23120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42459,127.0.98.61:39657,127.0.98.60:38865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:56.035770 23120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:56.036847 23120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:56.046036 23125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:56.046059 23126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:56.046059 23128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:56.046263 23120 server_base.cc:1034] running on GCE node
I20260924 04:45:56.046869 23120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:56.048220 23120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:56.049435 23120 hybrid_clock.cc:648] HybridClock initialized: now 1790225156049418 us; error 30 us; skew 500 ppm
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:56.052517 23120 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:45:56.053536 23120 webserver.cc:460] Webserver started at http://127.0.98.61:36667/ using document root <none> and password file <none>
I20260924 04:45:56.054179 23120 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:56.054278 23120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:56.054522 23120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:56.056773 23120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data/instance:
uuid: "83fe37f6309349229b83394b4aebb8ee"
format_stamp: "Formatted at 2026-09-24 04:45:56 on dist-test-slave-npg2"
server_key: "1f5aa3cd04c13bedfe51a56270a1c528"
server_key_iv: "8ed659dec87321fe07cb609e6eb18bdf"
server_key_version: "encryptionkey@0"
I20260924 04:45:56.057375 23120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal/instance:
uuid: "83fe37f6309349229b83394b4aebb8ee"
format_stamp: "Formatted at 2026-09-24 04:45:56 on dist-test-slave-npg2"
server_key: "1f5aa3cd04c13bedfe51a56270a1c528"
server_key_iv: "8ed659dec87321fe07cb609e6eb18bdf"
server_key_version: "encryptionkey@0"
I20260924 04:45:56.061618 23120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:45:56.064554 23135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:56.066170 23120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:45:56.066329 23120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal
uuid: "83fe37f6309349229b83394b4aebb8ee"
format_stamp: "Formatted at 2026-09-24 04:45:56 on dist-test-slave-npg2"
server_key: "1f5aa3cd04c13bedfe51a56270a1c528"
server_key_iv: "8ed659dec87321fe07cb609e6eb18bdf"
server_key_version: "encryptionkey@0"
I20260924 04:45:56.066460 23120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:56.094023 23120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:56.097823 23120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:56.098053 23120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:56.105369 23120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39657
I20260924 04:45:56.105676 23187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39657 every 8 connection(s)
I20260924 04:45:56.106459 23120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:45:56.109598 23188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:56.111989 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } attempt: 3
I20260924 04:45:56.114100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 23120
I20260924 04:45:56.114240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-1/wal/instance
I20260924 04:45:56.114567 392 external_mini_cluster.cc:1428] Setting key 357089e72eeb11c7d47b8f485a8bef02
I20260924 04:45:56.114213 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:45:56.135267 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.112558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41222)
0924 04:45:56.112749 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.112768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.112880 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.116184 (+ 3304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.116187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.116393 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.116554 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.116563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.118556 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.118563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.119584 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.119596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.120535 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.120538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.120700 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.128807 (+ 8107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.128825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.130672 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.133058 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.133063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.133066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.133333 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.133772 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.133778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.133782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.133902 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.134618 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.134621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.134734 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.134877 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.135049 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":30,"spinlock_wait_cycles":8960}
I20260924 04:45:56.136700 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.113416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.0.98.61:39657)
0924 04:45:56.113994 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.114017 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.114066 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:45:56.115303 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.115333 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.115822 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.116216 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.116736 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.118398 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.119712 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.120379 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.120575 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.130883 (+ 10308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.130893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.130910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.130941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.132928 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.133472 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.133478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.133484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.133668 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.134029 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.134032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.134034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.134487 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.134642 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.135057 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.135430 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":397,"thread_start_us":167,"threads_started":1}
I20260924 04:45:56.136961 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.116900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42459 (local address 127.0.0.1:37960)
0924 04:45:56.117277 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.117327 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.117483 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.117867 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.117872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.118203 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.118380 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.118390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.119293 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.119300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.120217 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.120227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.121173 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.121179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.121310 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.128915 (+ 7605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.128946 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.131410 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.134942 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.134957 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.134967 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.135333 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.135711 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.135715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.135720 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.135814 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.136266 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.136273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.136425 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.136575 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.136723 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260924 04:45:56.136958 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.116928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.0.98.62:42459)
0924 04:45:56.117052 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.117057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.117396 (+ 339us) server_negotiation.cc:407] Connection header received
0924 04:45:56.117590 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.117595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.117648 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.117706 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.118528 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.119170 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.120344 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.120847 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.120985 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.131666 (+ 10681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.131670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.131673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.131708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.134788 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.135466 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.135471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.135474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.135562 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.135921 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.135924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.135925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.136143 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.136242 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.136671 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.136802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":896}
I20260924 04:45:56.138751 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:56.139024 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:56.139861 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.139573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49880)
0924 04:45:56.139695 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.139785 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:45:56.140740 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.140501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49888)
0924 04:45:56.140653 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.140695 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 04:45:56.140893 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:45:56.146529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
/tmp/dist-test-taskhiQnGj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38865
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42459,127.0.98.61:39657,127.0.98.60:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:45:56.148685 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.139430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41234)
0924 04:45:56.139664 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.139681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.139780 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.140085 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.140087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.140279 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.140388 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.140394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.141330 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.141332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.141891 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.141899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.142576 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.142579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.142648 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.143224 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.143240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.144616 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.146864 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.146870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.146875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.147174 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.147466 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.147472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.147476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.147579 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.148042 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.148045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.148135 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.148244 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.148430 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260924 04:45:56.148685 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.139510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.0.98.61:39657)
0924 04:45:56.139641 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.139645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.139763 (+ 118us) server_negotiation.cc:407] Connection header received
0924 04:45:56.139860 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.139863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.139914 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.139994 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.140501 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.141215 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.142106 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.142468 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.142559 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.144740 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.144743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.144745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.144786 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.146538 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.147274 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.147277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.147278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.147364 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.147670 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.147672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.147673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.147936 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.148055 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.148288 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.148493 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":28}
I20260924 04:45:56.149937 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:56.150700 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.150431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49890)
0924 04:45:56.150566 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.150641 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 04:45:56.150995 23136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
I20260924 04:45:56.151929 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.151714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49894)
0924 04:45:56.151821 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.151867 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:45:56.152158 23188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
I20260924 04:45:56.172075 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } attempt: 1
I20260924 04:45:56.172978 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.172665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49904)
0924 04:45:56.172823 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.172897 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 04:45:56.173734 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.173520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49914)
0924 04:45:56.173632 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.173684 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:45:56.173897 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
I20260924 04:45:56.201431 23188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } attempt: 1
I20260924 04:45:56.202446 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.202063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49922)
0924 04:45:56.202258 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.202319 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 04:45:56.203243 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.202991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49928)
0924 04:45:56.203087 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.203181 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:45:56.203454 23188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
I20260924 04:45:56.249081 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } attempt: 2
I20260924 04:45:56.249996 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.249679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49938)
0924 04:45:56.249838 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.249905 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 04:45:56.250759 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.250545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49952)
0924 04:45:56.250654 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.250705 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:45:56.250895 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
W20260924 04:45:56.264685 23194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:45:56.265067 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:45:56.265132 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:45:56.265230 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:45:56.269554 23194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:45:56.269652 23194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:45:56.269688 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:45:56.269726 23194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:45:56.269765 23194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:45:56.271616 23188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } attempt: 2
I20260924 04:45:56.272480 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.272160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49968)
0924 04:45:56.272321 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.272396 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 04:45:56.273296 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.273051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49972)
0924 04:45:56.273173 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.273228 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:45:56.273455 23188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38865: connect: Connection refused (error 111)
I20260924 04:45:56.276264 23194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42459,127.0.98.61:39657,127.0.98.60:38865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:45:56.277444 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:45:56.278563 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:45:56.287395 23200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:56.287402 23201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:45:56.287475 23203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:45:56.287886 23194 server_base.cc:1034] running on GCE node
I20260924 04:45:56.288509 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:45:56.289670 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:45:56.290904 23194 hybrid_clock.cc:648] HybridClock initialized: now 1790225156290866 us; error 54 us; skew 500 ppm
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:45:56.294049 23194 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:45:56.294997 23194 webserver.cc:460] Webserver started at http://127.0.98.60:39477/ using document root <none> and password file <none>
I20260924 04:45:56.295677 23194 fs_manager.cc:359] Metadata directory not provided
I20260924 04:45:56.295784 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:45:56.296031 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:45:56.298185 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data/instance:
uuid: "03ccd6cf1b454bcd92a798c101363957"
format_stamp: "Formatted at 2026-09-24 04:45:56 on dist-test-slave-npg2"
server_key: "5867c559afe9f4e573c4f8036097ddff"
server_key_iv: "5eabb73a8da948e9fbdbd5830a86c816"
server_key_version: "encryptionkey@0"
I20260924 04:45:56.298766 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal/instance:
uuid: "03ccd6cf1b454bcd92a798c101363957"
format_stamp: "Formatted at 2026-09-24 04:45:56 on dist-test-slave-npg2"
server_key: "5867c559afe9f4e573c4f8036097ddff"
server_key_iv: "5eabb73a8da948e9fbdbd5830a86c816"
server_key_version: "encryptionkey@0"
I20260924 04:45:56.302584 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:45:56.305368 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:45:56.306668 23194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:45:56.306815 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal
uuid: "03ccd6cf1b454bcd92a798c101363957"
format_stamp: "Formatted at 2026-09-24 04:45:56 on dist-test-slave-npg2"
server_key: "5867c559afe9f4e573c4f8036097ddff"
server_key_iv: "5eabb73a8da948e9fbdbd5830a86c816"
server_key_version: "encryptionkey@0"
I20260924 04:45:56.306939 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:45:56.325842 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:45:56.329247 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:45:56.329464 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:45:56.336851 23194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38865
I20260924 04:45:56.337188 23262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38865 every 8 connection(s)
I20260924 04:45:56.337920 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:45:56.340819 23263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:45:56.344390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu as pid 23194
I20260924 04:45:56.344532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhiQnGj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224865297037-392-0/minicluster-data/master-2/wal/instance
I20260924 04:45:56.344808 392 external_mini_cluster.cc:1428] Setting key 724def7385c3decf59eed2294abdf7d5
I20260924 04:45:56.344910 23263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:56.350121 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } attempt: 3
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:45:56.359920 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.346854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.0.98.62:42459)
0924 04:45:56.346996 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.347003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.347328 (+ 325us) server_negotiation.cc:407] Connection header received
0924 04:45:56.348004 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.348007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.348065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.348148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.349266 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.350102 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.351487 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.351840 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.351936 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.355187 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.355194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.355199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.355230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.357092 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.357903 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.357906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.357908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.357994 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.358444 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.358449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.358451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.358815 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.358915 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.359506 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.359659 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:45:56.361179 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.346763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42459 (local address 127.0.0.1:37964)
0924 04:45:56.347183 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.347231 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.347706 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.348230 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.348237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.348755 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.349122 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.349134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.350225 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.350228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.351369 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.351377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.352096 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.352099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.352287 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.353021 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.353042 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.354966 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.357179 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.357189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.357201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.357781 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.358088 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.358093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.358096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.358291 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.359007 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.359015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.359228 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.360131 (+ 903us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.360340 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260924 04:45:56.361186 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.346160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.0.98.60:38865)
0924 04:45:56.346561 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.346576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.346608 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:45:56.347683 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.347711 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.347956 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.348148 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.349093 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.350528 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.351924 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.352285 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.352402 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.355593 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.355605 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.355616 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.355636 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.357749 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.358227 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.358231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.358232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.358298 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.358587 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.358592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.358594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.358976 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.359756 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.359873 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.360195 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260924 04:45:56.361491 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.350614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.0.98.60:38865)
0924 04:45:56.350889 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.350900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.350912 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:45:56.350941 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.350944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.350979 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.351068 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.351522 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.352302 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.353723 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.354031 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.354145 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.356766 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.356772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.356777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.356817 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.359493 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.359956 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.359961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.359966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.360048 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.360559 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.360563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.360565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.360793 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.360906 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.361208 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.361324 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
I20260924 04:45:56.361991 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.350560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:49984)
0924 04:45:56.350711 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.350729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.350802 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.351101 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.351104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.351328 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.351422 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.351428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.352432 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.352435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.353554 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.353565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.354160 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.354163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.354255 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.354746 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.354761 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.356374 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.359601 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.359605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.359610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.359832 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.360159 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.360163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.360165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.360249 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.360922 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.360925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.361022 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.361133 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.361838 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:45:56.362994 23263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:56.364682 23188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } attempt: 3
I20260924 04:45:56.366398 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08: Bootstrap starting.
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:45:56.370227 23113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08: Neither blocks nor log segments found. Creating new log.
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:45:56.372378 23113 log.cc:824] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:56.374519 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.364084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.0.98.62:42459)
0924 04:45:56.364171 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.364175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.364283 (+ 108us) server_negotiation.cc:407] Connection header received
0924 04:45:56.364448 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.364450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.364497 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.364593 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.365342 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.365949 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.366834 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.367179 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.367257 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.370695 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.370702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.370708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.370740 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.372446 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.372977 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.372983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.372985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.373074 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.373414 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.373418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.373420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.373655 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.373807 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.374187 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.374336 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:45:56.374617 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.363643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.0.98.61:39657)
0924 04:45:56.363767 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.363772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.363835 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:45:56.364003 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.364006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.364064 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.364149 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.364718 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.365514 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.366871 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.367202 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.367278 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.370812 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.370816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.370818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.370843 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.372735 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.373307 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.373311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.373313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.373394 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.373679 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.373684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.373685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.373937 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.374045 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.374324 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.374470 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:45:56.375283 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.363521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39657 (local address 127.0.0.1:41248)
0924 04:45:56.363778 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.363793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.363890 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.364278 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.364281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.364463 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.364578 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.364584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.365657 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.365661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.366446 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.366456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.367439 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.367442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.367568 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.369065 (+ 1497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.369079 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.370671 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.372920 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.372923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.372925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.373196 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.373488 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.373493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.373495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.373584 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.374036 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.374039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.374137 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.374908 (+ 771us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.375051 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":26}
I20260924 04:45:56.375631 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.365094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:50000)
0924 04:45:56.365327 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.365341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.365407 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.366814 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.366819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.367015 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.367111 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.367155 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.368115 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.368119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.368895 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.368906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.369539 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.369544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.369634 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.370118 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.370132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.371676 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.373626 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.373629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.373631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.373968 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.374278 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.374282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.374287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.374373 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.375086 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.375089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.375224 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.375312 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.375439 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 04:45:56.376021 23263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:45:56.376672 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.365590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.0.98.60:38865)
0924 04:45:56.366608 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.366612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.366622 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:45:56.366651 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.366653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.366690 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.366762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.367279 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.368011 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.369022 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.369395 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.369840 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.371825 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.371830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.371836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.371871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.373516 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.374074 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.374077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.374080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.374174 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.374494 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.374500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.374502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.374969 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.375906 (+ 937us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.376345 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.376469 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":955,"spinlock_wait_cycles":1920}
I20260924 04:45:56.377948 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08: No bootstrap required, opened a new log
I20260924 04:45:56.379803 23188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: Bootstrap starting.
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:45:56 dist-test-slave-npg2 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:45:56.382891 23188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:56.382697 23113 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.382977 23113 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e70cc6bcc146598faa05ee67fb8d08, State: Initialized, Role: FOLLOWER
I20260924 04:45:56.383368 23113 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.383903 23188 log.cc:824] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:56.385005 23113 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:56.386302 23188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: No bootstrap required, opened a new log
I20260924 04:45:56.386548 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.375804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.0.98.61:39657)
0924 04:45:56.375980 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.375986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.375997 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:45:56.376026 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.376029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.376098 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.376175 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.376910 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.377632 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.378638 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.378929 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.379024 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.382521 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.382525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.382527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.382553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.384259 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.384795 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.384799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.384801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.384882 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.385182 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.385185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.385187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.385406 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.385573 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.386282 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.386413 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 04:45:56.388187 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.376475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38865 (local address 127.0.0.1:50008)
0924 04:45:56.376788 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 04:45:56.376802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:45:56.376869 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:45:56.377129 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:45:56.377133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:45:56.377286 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:45:56.377360 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.377366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.378262 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.378265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.379105 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:45:56.379116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.379855 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.379858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:45:56.379939 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.380371 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:45:56.380385 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:45:56.383451 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:45:56.385652 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.385655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.385658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.385899 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.386199 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:45:56.386201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:45:56.386203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:45:56.386282 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:45:56.387682 (+ 1400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:45:56.387685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:45:56.387773 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 04:45:56.387874 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:45:56.388012 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260924 04:45:56.388752 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.376631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.0.98.60:38865)
0924 04:45:56.376715 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.376720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.376936 (+ 216us) server_negotiation.cc:407] Connection header received
0924 04:45:56.376961 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.376963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.376995 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.377048 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.377486 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.378155 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.379283 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.379657 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.379758 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.383749 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.383753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.383756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.383782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.385534 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.386007 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.386011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.386015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.386098 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.386444 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.386447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.386449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.387546 (+ 1097us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.387642 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.388511 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.388629 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:45:56.389321 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.389459 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.390373 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:45:56.391602 23087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c4e70cc6bcc146598faa05ee67fb8d08"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37972
I20260924 04:45:56.391815 23087 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:45:56.391968 23087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:45:56.392592 23282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:56.392668 23282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:56.393177 23087 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.394050 23087 leader_election.cc:288] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [CANDIDATE]: Term 1 election: Requested vote from peers 83fe37f6309349229b83394b4aebb8ee (127.0.98.61:39657), 03ccd6cf1b454bcd92a798c101363957 (127.0.98.60:38865)
I20260924 04:45:56.395471 23263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: Bootstrap starting.
I20260924 04:45:56.391805 23045 master_runner.cc:186] Error getting master registration for 127.0.98.61:39657: OK, instance_id { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" instance_seqno: 1790225156103334 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:45:56.396610 23045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:45:56.398311 23263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: Neither blocks nor log segments found. Creating new log.
I20260924 04:45:56.399686 23263 log.cc:824] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: Log is configured to *not* fsync() on all Append() calls
I20260924 04:45:56.399936 23188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.400327 23188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fe37f6309349229b83394b4aebb8ee, State: Initialized, Role: FOLLOWER
I20260924 04:45:56.400840 23188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.402338 23263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: No bootstrap required, opened a new log
I20260924 04:45:56.403097 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "83fe37f6309349229b83394b4aebb8ee"
I20260924 04:45:56.403352 23162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:56.404932 23162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e70cc6bcc146598faa05ee67fb8d08 in term 1.
I20260924 04:45:56.405169 23188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:56.405704 23064 leader_election.cc:302] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fe37f6309349229b83394b4aebb8ee, c4e70cc6bcc146598faa05ee67fb8d08; no voters:
I20260924 04:45:56.406040 23270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:45:56.406303 23270 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [term 1 LEADER]: Becoming Leader. State: Replica: c4e70cc6bcc146598faa05ee67fb8d08, State: Running, Role: LEADER
I20260924 04:45:56.406769 23270 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.406777 23263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.408754 23263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ccd6cf1b454bcd92a798c101363957, State: Initialized, Role: FOLLOWER
I20260924 04:45:56.409302 23263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } }
I20260924 04:45:56.409300 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e70cc6bcc146598faa05ee67fb8d08. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.409499 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:56.411741 23237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03ccd6cf1b454bcd92a798c101363957"
I20260924 04:45:56.412006 23237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:45:56.412067 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.396549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.0.98.61:39657)
0924 04:45:56.396736 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.396740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.397862 (+ 1122us) server_negotiation.cc:407] Connection header received
0924 04:45:56.398052 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.398055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.398105 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.398157 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.399207 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.400000 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.401244 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.401572 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.401649 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.403592 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.403596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.403601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.403632 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.405383 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.410617 (+ 5234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.410621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.410623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.410727 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.411045 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.411049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.411050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.411296 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.411430 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.411784 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.411913 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 04:45:56.413985 23237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e70cc6bcc146598faa05ee67fb8d08 in term 1.
I20260924 04:45:56.415030 23263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:45:56.417213 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.397496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.0.98.62:42459)
0924 04:45:56.397602 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.397606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.402648 (+ 5042us) server_negotiation.cc:407] Connection header received
0924 04:45:56.403755 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.403759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.403825 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.403881 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.405131 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.405925 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.408972 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.409307 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.409406 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.413384 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.413387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.413390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.413423 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.415310 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.415820 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.415823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.415825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.415923 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.416238 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.416242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.416243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.416470 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.416582 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.416926 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.417049 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:45:56.417246 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.419239 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.416296 23120 master_runner.cc:186] Error getting master registration for 127.0.98.60:38865: OK, instance_id { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" instance_seqno: 1790225156334742 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:45:56.421239 23120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 04:45:56.422009 23317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:56.422086 23317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:56.428987 23305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:45:56.429502 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:56.419640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.0.98.60:38865)
0924 04:45:56.419742 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.419746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.421321 (+ 1575us) server_negotiation.cc:407] Connection header received
0924 04:45:56.421540 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.421544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.421593 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.421642 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.422579 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.423315 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.424402 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:56.424644 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:56.424762 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:56.426215 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:45:56.426219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:45:56.426222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:45:56.426247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:45:56.427819 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.428249 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.428253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.428255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.428338 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:45:56.428637 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:45:56.428641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:45:56.428642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:45:56.428828 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:45:56.428977 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:56.429212 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:45:56.429331 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:45:56.429996 23305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 04:45:56.435837 23303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:45:56.435935 23303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:45:56.435948 23237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [term 1 FOLLOWER]: Refusing update from remote peer c4e70cc6bcc146598faa05ee67fb8d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:56.436658 23162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [term 1 FOLLOWER]: Refusing update from remote peer c4e70cc6bcc146598faa05ee67fb8d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:45:56.437453 23270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:56.437896 23270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:45:56.443305 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e70cc6bcc146598faa05ee67fb8d08. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.445858 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e70cc6bcc146598faa05ee67fb8d08. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.445977 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.446081 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.446871 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.446995 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.450790 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.450938 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:56.451215 23319 mvcc.cc:205] Tried to move back new op lower bound from 7332762240752390144 to 7332762240648056832. Current Snapshot: MvccSnapshot[applied={T|T < 7332762240752390144}]
I20260924 04:45:56.452489 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.452605 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.456966 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.457090 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08 [sys.catalog]: This master's current role is: LEADER
I20260924 04:45:56.457432 23322 mvcc.cc:205] Tried to move back new op lower bound from 7332762240752390144 to 7332762240648056832. Current Snapshot: MvccSnapshot[applied={T|T < 7332762240752390144}]
I20260924 04:45:56.458328 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e70cc6bcc146598faa05ee67fb8d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42459 } } peers { permanent_uuid: "83fe37f6309349229b83394b4aebb8ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39657 } } peers { permanent_uuid: "03ccd6cf1b454bcd92a798c101363957" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38865 } } }
I20260924 04:45:56.458452 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:45:56.458709 23324 mvcc.cc:205] Tried to move back new op lower bound from 7332762240752390144 to 7332762240648056832. Current Snapshot: MvccSnapshot[applied={T|T < 7332762240752390144}]
I20260924 04:45:56.469122 23305 catalog_manager.cc:1347] Generated new cluster ID: b4c419da5b064f26893208b539126f04
I20260924 04:45:56.469219 23305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:45:56.495226 23305 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:45:56.496418 23305 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:45:56.512074 23305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e70cc6bcc146598faa05ee67fb8d08: Generated new TSK 0
I20260924 04:45:56.513161 23305 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:45:56.545634 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.543530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.0.98.61:39657)
0924 04:45:56.543646 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.543650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.544419 (+ 769us) server_negotiation.cc:407] Connection header received
0924 04:45:56.544651 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.544655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.544709 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.544765 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.545489 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41284: BlockingRecv error: recv got EOF from 127.0.0.1:41284 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:45:56.547161 23268 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.542565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.0.98.60:38865)
0924 04:45:56.542716 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.542721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.546191 (+ 3470us) server_negotiation.cc:407] Connection header received
0924 04:45:56.546365 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.546368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.546419 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.546472 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.546990 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50028: BlockingRecv error: recv got EOF from 127.0.0.1:50028 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:45:56.548527 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.543243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.98.62:42459)
0924 04:45:56.543396 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.543402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.547642 (+ 4240us) server_negotiation.cc:407] Connection header received
0924 04:45:56.547804 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.547808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.547855 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.547905 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.548398 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37990: BlockingRecv error: recv got EOF from 127.0.0.1:37990 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:45:56.553400 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.551793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.0.98.61:39657)
0924 04:45:56.551906 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.551911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.552377 (+ 466us) server_negotiation.cc:407] Connection header received
0924 04:45:56.552599 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.552602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.552685 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.552740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.553290 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41296: BlockingRecv error: recv got EOF from 127.0.0.1:41296 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:45:56.561879 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.560120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.0.98.61:39657)
0924 04:45:56.560248 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.560255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.560828 (+ 573us) server_negotiation.cc:407] Connection header received
0924 04:45:56.561056 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.561060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.561119 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.561175 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.561763 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41306: BlockingRecv error: recv got EOF from 127.0.0.1:41306 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:45:56.568347 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.566505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.98.61:39657)
0924 04:45:56.566621 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.566625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.567228 (+ 603us) server_negotiation.cc:407] Connection header received
0924 04:45:56.567441 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.567445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.567510 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.567572 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.568226 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:45:56.576987 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.575188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.0.98.61:39657)
0924 04:45:56.575331 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.575336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.575953 (+ 617us) server_negotiation.cc:407] Connection header received
0924 04:45:56.576184 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.576188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.576241 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.576294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.576849 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41332: BlockingRecv error: recv got EOF from 127.0.0.1:41332 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 04:45:56.590587 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.588566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.0.98.61:39657)
0924 04:45:56.588712 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.588717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.589426 (+ 709us) server_negotiation.cc:407] Connection header received
0924 04:45:56.589672 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.589676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.589740 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.589802 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.590452 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41334: BlockingRecv error: recv got EOF from 127.0.0.1:41334 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:45:56.611308 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.609317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.0.98.61:39657)
0924 04:45:56.609448 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.609454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.610108 (+ 654us) server_negotiation.cc:407] Connection header received
0924 04:45:56.610362 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.610367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.610448 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.610510 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.611180 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41342: BlockingRecv error: recv got EOF from 127.0.0.1:41342 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:45:56.647302 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.645954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.0.98.61:39657)
0924 04:45:56.646088 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.646093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.646346 (+ 253us) server_negotiation.cc:407] Connection header received
0924 04:45:56.646526 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.646530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.646593 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.646686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.647180 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41346: BlockingRecv error: recv got EOF from 127.0.0.1:41346 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:45:56.714648 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.713475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.0.98.61:39657)
0924 04:45:56.713627 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.713631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.713733 (+ 102us) server_negotiation.cc:407] Connection header received
0924 04:45:56.713867 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.713870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.713965 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.714055 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.714545 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41350: BlockingRecv error: recv got EOF from 127.0.0.1:41350 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 04:45:56.846747 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:56.845357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.0.98.61:39657)
0924 04:45:56.845581 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:56.845586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:56.845754 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:45:56.845943 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:56.845947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:56.846012 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:56.846128 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:56.846625 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41354: BlockingRecv error: recv got EOF from 127.0.0.1:41354 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260924 04:45:57.106755 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:45:57.105174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.98.61:39657)
0924 04:45:57.105306 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:57.105310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:57.105512 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:45:57.105693 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:57.105697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:57.105753 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:57.105807 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:45:57.106626 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41362: BlockingRecv error: recv got EOF from 127.0.0.1:41362 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:45:57.423591 23317 catalog_manager.cc:1259] Loaded cluster ID: b4c419da5b064f26893208b539126f04
I20260924 04:45:57.423702 23317 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: loading cluster ID for follower catalog manager: success
I20260924 04:45:57.425379 23317 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: acquiring CA information for follower catalog manager: success
I20260924 04:45:57.426578 23317 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03ccd6cf1b454bcd92a798c101363957: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:45:57.437254 23303 catalog_manager.cc:1259] Loaded cluster ID: b4c419da5b064f26893208b539126f04
I20260924 04:45:57.437362 23303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: loading cluster ID for follower catalog manager: success
I20260924 04:45:57.439270 23303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: acquiring CA information for follower catalog manager: success
I20260924 04:45:57.440563 23303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83fe37f6309349229b83394b4aebb8ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:45:57.625679 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:57.621263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.0.98.61:39657)
0924 04:45:57.621530 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:57.621534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:57.621551 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:45:57.621688 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:57.621690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:57.621815 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:57.621930 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:57.622404 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.623003 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.624165 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.624395 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.624490 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:57.624817 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:57.625260 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:57.625355 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:57.625497 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 04:45:57.625552 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260924 04:45:57.633255 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:57.629152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.98.60:38865)
0924 04:45:57.629373 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:57.629377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:57.629390 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:45:57.629476 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:57.629478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:57.629593 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:57.629695 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:57.629937 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.630482 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.631535 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.631776 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.631863 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:57.632275 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:57.632770 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:57.632872 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:57.633073 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:45:57.633131 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260924 04:45:57.641364 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:57.636839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.0.98.60:38865)
0924 04:45:57.637023 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:57.637027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:57.637688 (+ 661us) server_negotiation.cc:407] Connection header received
0924 04:45:57.637895 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:57.637899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:57.637960 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:57.638069 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:57.638432 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.639210 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.640164 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.640511 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.640619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:57.640873 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:57.640962 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:57.641051 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:57.641142 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:45:57.641195 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 04:45:57.641989 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:57.637396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.0.98.62:42459)
0924 04:45:57.637998 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:57.638004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:57.638020 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:45:57.638118 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:57.638121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:57.638282 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:57.638404 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:57.638682 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.639346 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.640460 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.640718 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.640825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:57.641188 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:57.641576 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:57.641666 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:57.641782 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 04:45:57.641842 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":72,"threads_started":1}
I20260924 04:45:57.642287 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:45:57.636749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.98.61:39657)
0924 04:45:57.636881 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:45:57.636886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:45:57.637122 (+ 236us) server_negotiation.cc:407] Connection header received
0924 04:45:57.637396 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:45:57.637399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:45:57.637477 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:45:57.637534 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:45:57.637999 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.638750 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.640098 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:45:57.640357 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:45:57.640479 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:45:57.641188 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:45:57.641267 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:45:57.641311 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 04:45:57.642101 (+ 790us) server_negotiation.cc:299] Negotiation successful
0924 04:45:57.642168 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:45:57.645527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 23045
I20260924 04:45:57.652115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 23120
I20260924 04:45:57.658322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhiQnGj/build/debug/bin/kudu with pid 23194
2026-09-24T04:45:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3161 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6455 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292354 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST